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Abstract 


The  purpose  of  this  research  was  to  explore  COTS-based  systems  as  they  are 
acquired  by  the  Air  Force.  Current  guidance  related  to  the  acquisition  of  COTS-based 
systems  is  explored.  Based  upon  the  literature  reviewed,  the  research  targeted  the 
specific  area  of  acquisition  plans.  A  multiple  case  study  of  acquisition  plans  from  several 
COTS-based  systems  was  performed. 

Current  guidance  related  to  acquisition  plans  has  not  been  specifically  tailored  to 
COTS-based  systems.  The  results  of  the  analysis  of  the  COTS-based  systems  showed 
that  the  use  total  ownership  cost  (TOC)  and  cost  as  an  independent  variable  (CAIV) 
enabled  a  system  to  be  highly  successful.  The  use  of  TOC  combined  with  the  use  of 
CAIV  in  a  COTS-based  system  ensures  a  system  has  flexible  requirements.  This 
flexibility  will  lead  to  maintaining  or  lowering  costs  while  increasing  operational 
capabilities.  Additionally,  a  plan  for  upgrades  in  a  COTS-based  system,  that  includes 
TOC  and  CAIV,  provides  for  reduced  life  cycle  costs  while  allowing  for  system 
upgrades.  It  is  imperative  that  any  future  acquisition  guidance  related  to  COTS-based 
systems  includes  TOC,  CAIV  and  a  plan  for  upgrades. 


AN  IDENTIFICATION  AND  DISCUSSION  OF  KEY  SUCCESS  FACTORS  IN  THE 
ACQUISITION  OF  COMMERCIAL-OFF-THE-SHELF  (COTS-)  BASED  SYSTEMS 


I.  Introduction 


Overview 

In  recent  years,  the  DoD  has  experienced  shrinking  budgets.  Although  yearly 
budgets  have  been  smaller,  the  DoD  still  realizes  the  importance  of  acquiring  the  best 
possible  weapons  systems.  Commercial-off-the-shelf  (COTS)  items  have  been  offered  as 
a  means  for  DoD  programs  to  reduce  acquisition  costs  while  keeping  current  technology 
in  the  hands  of  the  warfighter.  In  COTS-based  systems,  commercial  hardware  and 
software  is  used  to  satisfy  the  needs  of  the  system. 

Best  Value.  Acquisition  professionals  determine  which  items  to  purchase  by 
performing  a  value  analysis.  A  value  analysis  is  the  relationship  between  value, 
attributes,  and  cost.  The  user  subjectively  determines  the  value  of  the  item.  The 
attributes  of  the  item  are  associated  with  the  product  or  service  itself.  Quality,  delivery, 
maintenance  and  ease  of  use  are  all  attributes  of  an  item.  As  an  equation,  the  value 
analysis  is  the  relationship  between  value,  attributes,  and  cost: 

Value  =  Attributes  of  the  Item  /  Cost  of  the  Item 
The  value  of  an  item  increases  when  the  cost  decreases  and  the  attributes  remain  the 
same.  The  value  of  an  item  can  also  increase  if  its  attributes  are  enhanced  while  the  cost 


1 


remains  the  same  or  is  lowered  (Monczka,  1998).  The  DoD  hopes  to  achieve  the  best 
value  when  acquiring  COTS-based  systems. 

Problems.  Recent  studies  have  shown  problems  with  COTS-based  systems. 
Although  costs  may  be  lower  in  the  development  stage,  unforeseen  sustainment  issues 
have  caused  total  life  cycle  costs  to  be  higher  than  traditionally  developed  systems.  In 
COTS-based  systems  these  life-cycle  costs  in  the  acquisition,  operation,  support,  and 
disposal  of  the  system  are  difficult  to  determine.  In  addition  to  the  cost  problems,  COTS- 
based  systems  have  different  risks  associated  with  them,  especially  with  regard  to 
interoperability.  Interoperability  is  the  ability  of  one  system  to  work  with  another  system 
(AFI  10-601, 1998).  COTS  products  are  developed  by  vendors  for  the  commercial 
marketplace  with  little  regard  for  the  military  system  in  which  they  are  included  (Tracz, 
2000).  Product  upgrades  are  also  developed  for  the  commercial  marketplace.  These 
upgrades  may  or  may  not  work  in  a  COTS-based  system.  Interoperability  of  the  COTS- 
based  system  is  risked  each  time  a  vendor  upgrades  its  product. 

Acquisition  Strategy.  To  overcome  these  problems,  a  recent  Air  Force  Scientific 
Advisory  Board  (SAB)  recommended  that  the  Air  Force  prepare  and  promulgate  policy 
with  regard  to  the  acquisition  strategy  used  for  COTS-based  systems.  An  acquisition 
strategy  is  developed  to  manage  the  acquisition  to  meet  the  user's  needs  within  resource 
constraints  (DoD  5000.2R,  1999).  The  acquisition  strategy  is  then  documented  in  the 
acquisition  plan,  which  is  required  for  all  acquisitions.  Currently,  Air  Force  guidance  on 
acquisition  plans  does  not  specifically  address  issues  with  COTS-based  systems. 

Due  to  the  development  process,  technology  cycle  time,  upgrade  issues,  and 
budget  differences.  Air  Force  policy  needs  to  address  the  strategy  used  in  acquiring  a 
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COTS-based  systems.  Typical  system  development  in  the  DoD  has  been  accomplished 
by  defining  the  need,  designing  the  item,  and  then  implementing  the  solution.  This  is 
known  as  waterfall  development.  COTS  systems  require  simultaneous  definition,  design, 
and  implementation  of  new  technology.  This  approach  is  called  spiral  development 
(Grant,  2000). 

COTS-based  systems  are  developed  in  order  to  take  advantage  of  the  most  current 
technology.  Military  systems,  built  to  last  20  years  or  more,  are  antiquated  by  technology 
that  can  change  every  18  months.  This  technology  cycle  time  creates  an  imbalance  that 
can  be  taken  advantage  of  by  using  COTS-based  systems.  Typically,  DoD  systems  do 
not  rely  on  the  marketplace  to  control  upgrades.  Changes  are  usually  determined  more 
by  the  system  designers  than  the  marketplace.  In  order  to  ensure  vendor  support, 
upgrades  need  to  be  made  to  COTS-based  systems.  These  changes,  which  are 
determined  by  the  marketplace,  can  affect  the  interoperability  of  COTS-based  systems. 
These  continuous  systems  upgrades  affect  the  operations  and  support  costs  of  COTS- 
based  systems.  The  development  processes,  technology  cycle  time,  upgrade  issues,  and 
budgetary  problems  in  COTS-based  systems  requires  the  development  of  new  acquisition 
plan  guidance. 

Problem  Statement 

COTS-based  systems  have  been  proposed  as  a  solution  to  budget  problems  in  the 
military.  However,  problems  such  as  life-cycle  cost  and  interoperability  can  reduce  the 
benefits  attained  by  COTS-based  systems.  COTS-based  systems  need  to  be  developed  in 
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a  spiral  approach  rather  than  a  waterfall  approach.  They  also  need  to  have  continuous 
upgrades  as  determined  by  the  commercial  marketplace.  These  upgrades  lead  to 
increased  interoperability  problems  in  COTS-based  systems.  In  order  to  attain  the 
maximum  benefits  from  COTS-based  systems,  their  acquisition  plans  need  to  be  tailored 
specifically  to  COTS-based  systems.  This  leads  to  the  specific  problem  statement: 
Currently,  there  is  no  standardized  guidance  for  the  development  of  an  acquisition 
plan  for  a  COTS-based  system. 

There  is  no  guidance  or  model  of  an  acquisition  plan  specifically  tailored  to  the 
acquisition  of  COTS-based  systems.  DOD  5000. 2R,  Mandatory  Procedures  for  Major 
Defense  Acquisition  Programs  and  Major  Automated  Information  Systems  Acquisition 
Programs,  provides  guidance  for  developing  acquisition  strategy.  A  plethora  of 
information  is  available  for  DoD  acquisition  professionals  to  use  in  developing 
acquisition  strategies.  However,  the  DoD  has  only  recently  published  considerations  and 
lessons  learned  for  COTS-based  systems.  This  still  does  not  provide  specific  guidance 
for  developing  an  acquisition  strategy  for  COTS-based  systems.  Acquisition 
professionals  do  not  have  a  reference  to  use  in  developing  acquisition  plans  for  COTS- 
based  systems.  This  study  examines  the  acquisition  plans  used  in  COTS-based  systems 
and  provides  recommended  guidance  in  the  acquisition  plans  for  these  systems. 
Specifically,  this  research  focuses  on  how  acquisition  plans  affects  the  success  of  COTS 
based-systems. 
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Research  Objective 


In  order  to  supply  a  solution  to  the  problem  statement,  two  research  objectives 
were  identified.  The  first  research  objective  was  to  develop  key  success  factors  to  be 
included  in  the  acquisition  plans  of  COTS-based  systems.  The  second  research  objective 
was  to  identify  which  critical  items  need  to  be  included  in  the  acquisition  plans  of  COTS- 
based  systems.  Reaching  these  objectives  should  enable  the  development  of  acquisition 
plan  guidance. 

Research  Questions 

To  develop  the  guidance  for  COTS-based  systems  acquisition  plans,  key  success 
factors  in  the  acquisition  plans  of  successful  COTS-based  systems  were  identified.  The 
key  success  factors  were  reviewed  to  determine  how  they  impacted  program  success. 
Critical  factors  were  also  developed  and  reviewed.  Additionally,  the  quantity  of  critical 
factors  was  reviewed  to  determine  if  not  one,  but  a  combination  of  common  items  led  to 
success.  The  problem  statement  was  investigated  by  addressing  these  questions: 
Research  Question  1 .  "Is  there  a  relationship  between  key  factors  of  an 
acquisition  plan  and  highly  successful  programs?" 

Research  Question  2  "How  do  the  key  success  factors  affect  success  of  the 
program?" 

Research  Question  3.  "How  many  critical  items  need  to  be  present  in  the 
acquisition  plan  for  the  program  to  be  rated  highly  successful  by  the  SAB?" 
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Methodology 


In  answering  the  research  questions,  real  world  case  studies  were  used  to  analyze 
the  COTS-based  programs  rated  highly  successful  and  COTS-based  programs  not  rated 
highly  successful.  Current  literature  was  reviewed  to  identify  key  factors.  The 
acquisition  plans  from  these  case  studies  were  then  compared  to  determine  which  factors 
led  to  the  program  being  rated  highly  successful.  Once  these  items  were  identified,  they 
were  studied  to  see  how  they  affected  the  success  of  the  program.  Additionally,  critical 
items  were  identified  from  the  key  success  factors.  The  critical  items  were  viewed 
cumulatively  to  determine  if  a  certain  number  of  critical  item  in  an  acquisition  plan  leads 
to  program  success. 

Scope 

This  research  effort  examined  the  COTS-based  systems  identified  in  the  Air  Force 
SAB  report  entitled  Ensuring  Successful  Implementation  of  Commercial  Items  in  Air 
Force  Systems.  The  SAB  studied  34  different  COTS-based  systems  to  develop  a 
checklist  of  actions  that  need  to  take  place  to  ensure  the  successful  integration  of  COTS 
into  Air  Force  systems  (Grant,  2000).  While  the  SAB  provided  a  checklist  of  items,  this 
study  attempts  to  determine  which  factors  are  most  important  to  the  success  of  a  system 
and  if  a  certain  number  of  factors  present  leads  to  a  highly  successful  system.  One  of  the 
recommendations  of  the  SAB  was  to  prepare  a  policy  to  drive  acquisition  strategy  of 
COTS-based  system.  Acquisition  plans  from  five  COTS-based  programs  rated  highly 
successful  and  from  five  programs  that  were  not  rated  highly  successful  were  reviewed. 
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Since  the  SAB  researched  only  military  systems,  this  research  was  also  specific  to 
military  systems  and  did  not  take  into  consideration  COTS-based  systems  outside  of  the 
DoD.  However,  the  ten  systems  studied  did  have  many  different  types  of  applications 
from  information  systems  to  guidance  kits  for  munitions.  The  research  focused  on  the 
acquisition  strategy  plans  as  outlined  in  the  Air  Force  Single  Acquisition  Management 
Plan  (SAMP)  Guide  (Guide,  1996).  The  acquisition  plans  were  studied  to  determine 
which  items  may  have  affected  program  success. 

Organization  of  Thesis 

This  chapter  provided  background  information  regarding  COTS.  Chapter  2, 
Literature  Review,  supplies  more  detailed  background  information  about  COTS-based 
systems  and  reasons  why  this  thesis  is  needed.  Chapter  3,  Methodology,  presents  the 
process  for  gathering  and  analyzing  the  data  and  supports  the  method  used.  Chapter  4, 
Analysis  of  Findings,  shows  the  results  of  the  data  gathering  and  provides  an  analysis  of 
that  data.  Chapter  5,  Summary  of  Findings,  presents  recommendations  and  conclusions 
based  on  the  analysis  of  findings. 
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II.  Literature  Review 


Overview 

This  chapter  provides  a  basis  of  knowledge  from  which  the  research  questions  can 
be  answered.  The  chapter  begins  by  defining  COTS-based  systems  and  explaining  why 
COTS-based  systems  are  used.  Following  this,  problems  associated  with  using  COTS- 
based  systems  are  explored.  The  chapter  then  explores  the  risks  related  to  problems  with 
COTS-based  systems.  The  means  of  overcoming  these  risks  are  then  examined.  After 
this,  the  chapter  addresses  acquisition  strategy.  Acquisition  strategy  is  defined  and 
available  guidance  for  acquisition  professionals  is  explored.  Next,  reasons  for  this 
research  are  provided  and  the  key  factors  and  critical  items  are  explained.  Finally,  the 
chapter  concludes  by  stating  the  need  for  further  studies  in  acquisition  strategy  of  COTS- 
based  systems. 

Background 

One  of  the  basic  questions  regarding  the  acquisition  of  a  COTS  based  system  is, 
What  constitutes  a  COTS-based  system?  A  simplified  answer  to  this  question  is  any 
system  that  uses  COTS-items.  However,  most  systems  often  do  contain  some  amount  of 
COTS  items.  The  difference  with  COTS-based  systems  now  is  the  wide  availability  of 
commercial  items  and  the  need  to  increase  their  use  in  DoD  systems  to  provide  the 
warfighter  with  the  latest  technological  advantage  (Albert  and  Morris,  2000).  As  defined 
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in  the  Guidelines  for  Successful  Acquisition  and  Management  of  Software-Intensive 
Systems,  a  COTS  item  is  one  which  has  been  developed,  produced,  and  tested  to  military 
or  commercial  standards  and  specifications  to  environmental  conditions  equal  to  or 
exceeding  those  required  by  the  weapon  system.  Additionally,  the  Guidelines  For 
Successful  Acquisition  and  Management  of  Software  Intensive  Systems  states  that  a 
COTS-item  is  readily  available  for  delivery  from  an  industrial  source  and  may  be 
acquired  without  charge  (Guidelines,  2000).  This  definition  then  begets  questions  about 
COTS-based  systems. 

Types  of  COTS-Based  Systems.  COTS-based  systems  are  easily  defined; 
however,  there  are  different  types  of  COTS-based  systems.  Simply  put,  a  COTS-based 
system  is  one  that  contains  components  that  are  COTS  products  (Clapp,  1998).  The 
different  types  of  COTS-based  systems  fall  on  a  continuum.  At  one  end  of  the  continuum 
are  COTS-solution  systems.  These  systems  are  a  single  product,  provided  by  one  vendor, 
that  provides  for  the  users  needs.  An  example  of  this  is  a  computer  program  that 
provides  all  the  needs  of  the  user.  On  the  other  end  of  the  continuum,  COTS-aggregate 
systems  are  made  up  of  many  COTS  product  from  many  different  vendors  that  are 
integrated  together  to  fulfill  the  users  need  (Brownsword,  2000).  This  is  like  a  custom- 
made  computer  bought  from  a  small  computer  store.  Still  other  COTS-based  systems,  in 
the  middle  of  the  continuum,  will  integrate  some  commercial  items  within  a  military 
developed  system  (Albert  and  Morris,  2000).  This  definition  of  COTS-based  systems 
provides  a  starting  point  to  examine  the  reasons  to  use  COTS-based  systems. 

Benefits.  Multiple  benefits  can  be  attained  when  acquiring  a  COTS-based 
system.  A  recent  Air  Force  Scientific  Advisory  Board  (SAB)  report  stated  “taking 
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advantage  of  COTS  products  seems  like  a  logical  way  to  achieve  significant  cost  savings 
with  very  little  sacrifice”(Grant,  2000:1).  A  Pentium-class  microprocessor  costs  between 
$250M  and  $400M  to  develop,  and  development  costs  are  escalating  at  nearly  40%  per 
year.  The  Air  Force  can  not  afford  expenses  of  this  magnitude  and  therefore  must  use 
COTS  (Grant,  2000). 

COTS-based  systems  also  allow  the  military  to  quickly  incorporate  new 
technology  into  weapons  systems  (Alford,  2000).  This  rapid  insertion  of  new  technology 
is  made  possible  in  COTS-based  systems  by  using  open  systems  architecture.  Open 
systems  adhere  to  commercial  interface  standards  and  are  easily  upgraded.  This  can  be 
compared  to  plug  and  play  components  in  personal  computers  (Obemdorf,  1998).  A 
military  advantage  goes  to  the  nation  that  captures  the  best  commercially  available 
technologies,  incorporates  them  in  weapons  systems,  and  gets  them  fielded  first 
(Hanratty,  1999).  The  Air  Force  Airborne  Warning  and  Control  System  (AWACS) 
computer  modernization  acquisition  used  an  open  system  architecture.  By  using  open 
systems,  future  upgrades  and  new  mission  capabilities  may  be  integrated  with  minimal 
integration  and  testing  requirements  (Milligan,  2000). 

Other  potential  benefits  are  lower  life  cycle  costs,  greater  reliability  and 
availability,  and  increased  support  from  the  industrial  base  (Albert  and  Morris,  2000). 
Life  cycle  costs  are  the  costs  attributable  to  acquisition,  operation,  support,  and  disposal 
of  a  system  (FAR,  2000).  Lower  life  cycle  costs  in  COTS-based  systems  come  from 
decreased  development  costs  during  acquisition.  In  the  commercial  marketplace,  a 
competitive  advantage  goes  to  the  companies  that  provide  items  with  the  best  value.  Part 
of  measuring  best  value  is  reliability  and  availability.  Companies  that  make  highly 


to 


reliable  items  available  in  the  marketplace  will  have  a  competitive  advantage.  Those 
companies  will  be  selected  as  vendors  for  COTS  items.  As  communications  and 
transportation  have  improved,  the  number  of  vendors  available  to  provide  support  for 
government  contracts  has  increased.  Support  from  the  industrial  base  increases  because 
more  companies  will  be  able  to  provide  support,  not  just  the  government  contractor. 
Using  COTS-based  systems  to  achieve  these  benefits  is  best  summed  up  by  Obemdorf 
and  Carney,  “In  systems  where  the  use  of  existing  commercial  components  is  both 
plausible  and  feasible,  it  is  no  longer  acceptable  for  the  government  to  specify,  build,  and 
maintain  a  large  array  of  comparable  proprietary  products”  (Obemdorf  and  Carney,  1998: 
1). 

Regulatory  Guidance.  Due  to  the  benefits  of  COTS-based  systems,  regulations 
now  mandate  their  use  when  possible.  The  Federal  Acquisition  Regulation  (FAR) 
applies  to  all  Department  of  Defense  purchases.  FAR  Part  12.101  calls  for  market 
research  to  be  done  and  states  “agencies  shall  acquire  commercial  items  or  non- 
developmental  items  when  they  are  available  to  meet  the  needs  of  the  agency"  (FAR, 
2000:12.101).  DOD  Directive  5000.1,  which  applies  to  all  DOD  acquisition  program, 
states  that  the  use  of  commercial  items  in  DOD  systems  is  the  preferred  approach  for 
meeting  operational  requirements  (Albert  and  Morris,  2000).  The  FAR,  along  with  DOD 
Directive  5000.1,  ensures  that  COTS  items  will  be  purchased  and  used  in  military 
systems.  Even  though  complex  defense  systems  may  not  be  manufactured  as  end  items 
on  commercial  lines,  their  subsystems  and  components  may  well  be  (Grant,  2000). 
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Problems 


Organizations  attempt  to  incorporate  new  technology  and  reduce  development 
cost  by  integrating  COTS  items.  Although  various  benefits  can  be  realized  when  using 
COTS-based  systems,  problems  have  been  encountered  in  their  use  (Holmes,  2000). 
Inflexible  requirements,  technology  cycle  time,  upgrades,  and  budgetary  problems  in 
COTS-based  systems  can  lead  to  system  failure. 

Inflexible  Requirements.  The  biggest  pitfall  of  all  in  COTS  systems  is  inflexible 
requirements.  If  the  COTS  needs  to  be  changed  to  meet  requirements,  the  cost  and 
schedule  reductions  disappear  (Grant,  2000).  Cost  and  schedule  reductions  are  achieved 
through  lack  of  product  development.  When  the  COTS  item  is  changed,  product 
development  takes  place,  erasing  some  of  these  benefits.  To  compound  the  problem, 
vendors  may  not  offer  support  for  items  that  have  been  modified. 

Technology  Cycle  Time.  The  amount  of  time  it  takes  for  technological 
advancements  to  be  designed,  developed,  and  fielded  -  technology  cycle  time  -  also 
causes  problems  in  COTS-based  systems  (Gillis,  1999).  The  life  of  a  typical  military 
acquisition  exceeds  20  years  (Alford,  2000).  The  development  time  for  new  DOD  major 
systems  is  between  8  and  15  years.  The  commercial  marketplace  has  drastically  faster 
technology  cycle  times.  These  include  computer  technology  with  a  cycle  time  of  1 8 
months,  6  years  for  avionics,  and  14  years  for  aircraft  engines  (Gillis,  1999).  If  the 
development  of  a  new  system  takes  8  years  and  the  life  of  the  system  is  20  years, 
computer  technology  will  have  changed  1 8  times  since  inception  of  the  system.  The 
system  will  more  than  likely  be  outdated  in  this  time  frame.  According  to  Kurt  Wallnau 
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of  the  Software  Engineering  Institute,  you  need  to  plan  on  evaluating  a  new  version  of  a 
COTS  product  every  six  months  (Tracz,  2000). 

Upgrades.  In  COTS-based  systems,  the  manufacturer  is  free  at  any  time  to  make 
changes  or  even  discontinue  the  manufacture  of  the  COTS  item  without  notice.  When 
these  changes  affect  the  form,  fit,  or  function  of  an  item,  it  can  cause  significant  problems 
with  the  COTS-based  system.  Upgrades  to  items  may  not  work  with  the  COTS  system 
and  replacements  may  not  be  available  (Alford,  2000).  Integration  of  various  commercial 
items  also  causes  problems  with  COTS.  As  the  number  of  COTS  components  and  COTS 
vendors  increase,  the  interplay  among  them  becomes  more  complex.  In  the  event  of 
system  failure,  it  may  be  difficult  to  prove  which  vendors  product  is  really  at  fault.  At  a 
minimum,  system  integrators  will  struggle  with  ways  to  keep  abreast  of  current 
technology  and  which  products  best  suit  their  needs  (Tracz,  2000).  These  are  not  the 
only  problems  that  plague  COTS-based  systems.  COTS-based  systems  can  also  have 
significant  budgetary  problems. 

Budget.  Budgetary  problems  in  a  COTS-based  system  come  from  the  incorrect 
application  of  life  cycle  costs.  COTS  components  provide  immediate  solutions  at  a  fixed 
cost.  However,  since  most  components  will  be  upgraded  during  the  life  of  the  system,  it 
is  unrealistic  to  assume  that  support  costs  will  be  zero  (Tracz,  2000).  Figure  1  shows  that 
the  cost  of  operations  and  support  is  almost  three  fourths  of  a  typical  system  (Alford, 
2000).  In  some  instances,  total  life  cycle  costs  of  COTS-based  systems  have  been  greater 
than  they  would  have  been  using  a  traditional  approach  (Grant,  2000).  Operations  and 
support  costs  of  COTS-based  systems  are  high  due  to  continuous  upgrade  of  items  and 
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Figure  1.  Typical  Cost  Distribution  (Alford,  2000:  13) 
changing  logistics  needs  for  these  upgraded  items.  However,  program  guidance  and 

budget  direction  does  not  reflect  the  need  for  greater  sustainment  costs  (Clapp,  1998). 
These  problems  with  COTS-based  systems  can  be  directly  linked  to  the  risks  associated 
with  acquiring  the  systems. 


Risks 


As  evidenced  by  the  problems  mentioned  above,  some  level  of  risk  is  involved  in 
acquiring  COTS-based  systems.  These  risks  involve  software/hardware  upgrade,  quality, 
security,  and  funding. 

Upgrade.  Failing  to  upgrade  to  the  latest  version  of  software/hardware  can  result 
in  loss  of  vendor  support  for  prior  versions  and  the  inability  to  buy  new  copies  or  obtain 
additional  copies  of  the  version  that  is  in  place.  Imagine  trying  to  upgrade  a  computer 
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from  DOS®  to  Windows®  ME  without  having  had  all  of  the  upgrades  in  between.  It 
might  just  be  easier  to  erase  the  hard  drive  and  install  a  full  version  of  Windows®  ME. 
Conversely,  upgrading  to  the  latest  version  can  result  in  the  new  version  being 
incompatible  with  the  rest  of  the  system,  increased  consumption  of  time  or  memory,  and 
operational  capabilities  of  the  system  which  may  not  be  fully  supported  (Clapp,  1998). 
When  installing  the  upgrades  from  DOS®  to  Windows®,  some  DOS  based  programs 
such  as  Enable  might  not  work  on  the  system  anymore.  Additionally,  new  hardware 
might  need  to  be  added  to  the  system  to  ensure  the  software  can  run. 

Quality.  Quality  of  a  COTS-based  system  is  risked  because  quality  is  a  subjective 
measure  depending  on  the  supplier's  point  of  view.  Traditional  systems  are  designed  to 
military  specifications  with  quality  being  one  of  the  criteria.  The  quality  of  traditional 
systems  is  assured  by  manufacturing  oversight  and  design  reviews.  In  a  COTS-based 
system,  the  DoD  looses  the  ability  to  provide  design  specifications  and  oversee  the 
manufacture  of  items.  Quality  of  an  item,  especially  an  upgraded  item,  may  not  be 
sufficient  for  exacting  military  systems.  This  can  be  especially  troublesome  problem 
with  software,  since  vendors  typically  fix  problems  in  the  next  version  of  the  product 
(Tracz,  2000).  If  an  upgraded  item  is  installed  in  a  system,  it  may  cause  the  whole 
system  to  shut  down.  Therefore,  new  versions  of  COTS  items  must  be  tested  before 
insertion  in  the  system  (Clapp,  2000). 

Security.  Security  risks  also  present  a  problem  with  COTS-based  systems. 
According  to  the  DoD  Year  2000  Management  Plan,  primary  vendors  may  have 
subcontractors  who  use  additional  subcontractors  that  employ  foreign  nationals  to  do  the 
actual  coding  of  the  COTS  (Year,  1998).  This  makes  COTS  software  especially 
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susceptible  to  a  trap  door  or  “Trojan  Horse”  (Grant,  2000).  A  trap  door  is  a  hidden 
software  or  hardware  mechanism  that  permits  systems  protection  mechanisms  to  be 
circumvented.  A  Trojan  Horse  is  a  computer  program  with  an  apparently  useful  function 
that  contains  hidden  functions  that  surreptitiously  exploit  the  legitimate  authorizations  of 
the  invoking  process  to  the  detriment  of  security.  A  computer  virus  is  a  form  of  a  Trojan 
Horse  (DoD  5200.28,  1999).  When  buying  a  specialized  piece  of  COTS  hardware,  there 
will  usually  be  software  embedded  in  the  equipment  (Vigder,  2000).  Therefore,  COTS 
hardware  and  software  are  both  susceptible  to  security  problems. 

Funding.  Funding  also  provides  some  risk  in  COTS-based  systems.  COTS-based 
systems  have  all  the  funding  risks  of  traditional  systems  and  more.  The  uncertainty  of 
product  upgrades,  coupled  with  changes  that  may  need  to  be  made  to  the  rest  of  the 
system,  make  it  difficult  to  estimate  proper  funding  requests  (Clapp,  1998).  Cost  models 
for  COTS  can  be  helpful,  but  the  development  of  new  publicly  available  COTS  cost 
estimation  techniques  and  models  is  still  in  its  infancy  (Brownsword,  2000).  With  all  of 
these  risks,  COTS-based  systems  would  never  succeed  if  there  were  no  means  of  risk 
reduction. 

Risk  Reduction 

Overcoming  the  risks  inherent  in  a  COTS-based  system  requires  a  paradigm  shift 
in  system  acquisition,  use  of  commercial  practices,  better  configuration  management,  and 
the  right  vendor. 
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System  Acquisition.  The  paradigm  shift  to  overcome  is  from  developing  a 
specific  product  for  a  specific  system  requirement  to  adjusting  specifications  to  what  the 
commercial  marketplace  has  to  offer.  In  COTS-based  systems,  requirements  of  the 
system  must  change  to  meet  the  ability  of  products  available  commercially.  The 
marketplace  drives  the  implementation  of  the  commercial  item;  therefore,  it  is  imperative 
to  know  the  fundamental  differences  between  integrating  commercial  items  and 
developing  a  custom  capability  (Albert  and  Morris,  2000). 

Commercial  Practices.  Programs  are  more  effective  when  adopting  commercial 
practices.  Understanding  the  nature  of  the  commercial  marketplace  will  help  reduce  risk 
associated  with  COTS  solutions  (Task  Order  054,  1999).  As  vendors  need  to  adapt  to  the 
government  bureaucracy,  procurement  organizations  will  see  costs  rise  (Albert  and 
Morris,  2000).  Therefore,  if  the  DOD  is  acquiring  a  COTS-based  system,  it  needs  to  do 
business  in  a  more  commercial  manner  (Brownsword,  2000). 

Configuration  Management.  Another  means  of  reducing  risk  is  good 
configuration  management.  Since  COTS  items  seldom  fit  together  well  with  other 
system  components,  adaptation  is  needed  to  make  the  items  fit  together  (Brownsword, 
2000).  Configuration  management  consists  of  tracking  which  versions  of  upgrades  are 
available  from  the  vendors,  which  are  installed,  and  at  which  sites  (Vigder,  1996). 

Correct  Vendor.  Identifying  the  best  contractor  can  also  lead  to  risk  reduction. 
Typically,  contractors  have  not  been  selected  for  their  ability  to  integrate  items, 
knowledge  of  the  marketplace,  or  expertise  with  specific  commercial  items.  In  COTS- 
based  systems,  these  factors  will  be  as  significant  as  traditional  factors  in  source  selection 
(Albert  and  Morris,  2000).  DOD  organizations  must  also  take  into  account  stability  of 
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the  vendor  and  willingness  to  work  with  the  DOD  as  part  of  the  acquisition 
(Brownsword,  2000).  While  these  efforts  can  reduce  the  risk  associated  with  acquiring 
COTS-based  systems,  a  strategy  is  needed  for  their  implementation. 

Strategy 

Definition.  An  acquisition  strategy  provides  direction  for  acquiring  a  system  from 
program  initialization  through  post-production  support.  The  primary  goal  of  developing 
an  acquisition  strategy  is  to  minimize  the  time  and  cost  to  satisfy  a  user’s  acquisition 
needs.  The  acquisition  strategy  addresses  such  issues  as  open  systems,  sources,  risk 
management,  cost  as  an  independent  variable,  contract  approach,  management  approach, 
environmental  considerations,  warranty  considerations,  and  sources  of  support. 
Acquisition  strategy  is  tailored  to  meet  the  needs  of  the  individual  program  (DoD  5200.2- 
R,  1999).  Development  of  the  acquisition  strategy  is  part  of  acquisition  planning  (FAR, 
2000). 

Guidance.  Available  guidance  in  the  DoD  states  that  all  acquisitions  should 
promote  and  provide  for  acquisition  of  commercial  items  (FAR,  2000).  However, 
guidance  on  the  acquisition  strategy  of  commercial  items  and  COTS-based  systems  is 
lacking.  DoD  5000.2-R  requires  that  contractors  incorporate  commercial  items  as 
components  of  items  supplied.  It  further  states  that  commercial  items  selected  shall  be 
based  on  open  systems  and  commercial  item  descriptions  to  the  maximum  extent 
practicable  (DoD  5200.2-R,  1999).  While  this  guidance  allows  for  flexibility  and 
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creativity  in  acquiring  COTS-based  systems,  it  does  not  provide  management  with 
enough  direction  to  ensure  a  COTS-based  system  will  have  an  adequate  acquisition  plan. 

The  available  guidance  on  COTS-based  acquisitions  is  limited.  The 
Software  Engineering  Institute  (SEI)  at  Carnegie  Mellon  University  has  produced  two 
documents  that  provide  lessons  learned  in  regards  to  COTS-based  systems,  Lessons 
Learned  Applying  Commercial  Off-the-Shelf  Products  and  Commercial  Items 
Acquisition:  Considerations  and  Lessons  Learned.  Neither  of  these  documents 
specifically  address  acquisition  plans.  However,  the  SEI  has  published  an  article  called 
An  Activity  Framework  for  COTS-Based  Systems  (Brownsword,  2000).  In  this  article, 
Brownsword,  Obemdorf,  and  Sledge  identify  nine  activities  to  help  develop  acquisition 
strategy  for  COTS-based  systems  (Brownsword,  2000:1 1): 

1 .  Identify  COTS-based  system  goals,  constraints  and  assumptions. 

2.  Identify  COTS-related  risks. 

3.  Identify  relevant  market  segments. 

4.  Identify  alternative  COTS-based  solutions. 

5.  Reassess  COTS-based  system  strategy  as  necessary. 

6.  Assess/evaluate/tradeoff  alternative  COTS-based  solutions. 

7.  Recommend  an  overall  COTS-based  system  strategy. 

8.  Create  a  corresponding  COTS-based  system  plan,  including  contingency  plans. 

9.  Reassess  and  revise  COTS-based  system  strategy  as  necessary. 

While  this  information  is  integral  to  building  an  acquisition  plan,  more  information  is 
needed  in  the  specific  areas  of  the  acquisition  plan. 


Research 


Acquisition  professionals  need  more  guidance  in  the  specific  areas  of  open 
systems,  sources,  risk  management,  cost  as  an  independent  variable,  contract  approach, 
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management  approach,  environmental  considerations,  warranty  considerations,  and 
sources  of  support  to  properly  develop  an  acquisition  strategy.  This  is  why  the  Scientific 
Advisory  Board  recommended  “that  the  Air  Force  prepare  and  promulgate  an 
implementation  policy  for  the  acquisition  and  sustainment  of  COTS-based  systems.  This 
policy  should  drive  acquisition  strategy. .  .’’(Grant,  2000).  In  order  to  gain  knowledge  of 
acquisition  strategy,  this  study  reviewed  acquisition  plans  of  COTS-based  systems  rated 
both  highly  successful  and  not  rated  highly  successful.  The  AFSAB  study  provided 
examples  of  both  types  of  programs.  In  order  to  eliminate  researcher  bias,  this  study  was 
limited  to  those  programs  identified  in  the  SAB  study. 

Identification  of  Key  Factors.  In  determining  what  to  include  in  an  acquisition 
plan,  potential  key  success  factors  were  identified.  The  following  questions  were  used  to 
study  the  potential  key  success  factors  in  the  acquisition  plans  of  COTS-based  systems. 
They  were  derived  by  taking  inputs  from  recommendations  contained  in  articles  on 
COTS-based  systems  and  reviewing  the  Single  Acquisition  Management  Plan  Guide. 
Italicized  questions  are  additionally  identified  as  critical  factors  for  use  in  research 
question  3.  Critical  factors  were  recommended  to  be  included  in  COTS-based  systems  by 
more  than  one  source.  Table  1  presents  the  questions  with  the  source(s)  that 
recommended  their  use. 

1 .  Are  the  requirements  flexible ? 

Albert  and  Morris  state  that  requirements  must  be  flexible  and  negotiable  (Albert  and 
Morris,  2000).  Both  NASA  and  SEI  have  emphasized  the  need  to  adapt  operational 
requirements  to  the  availability  of  the  COTS  components  (Vigder,  1996).  A  yes  will  be 
given  if  the  acquisition  plan  states  that  requirements  are  flexible.  Additionally,  a  yes 


20 


Carney  and 

Gant  and 

BimnswDnd  and 

Albert  and 

BrcwnsAod 

Cbemdorf 

Others 

Race 

Mem's 

and  others 

Question 


Mission 

Are  the  requirements  flexible?  X  X  X  X  X 

Program  Content 

Does  the  system  irterfarewth  other  progHi^?  X 


Is  this  a  joint  progarrf? 

Does  the  system  need  to  be  certified  before  being  put  into 
operation? 

Is  system  certification  done  by  the  mlitary? 

Acquisition  Strategy 

Is  the  &&D  oontract  Cost  Plus? 

Is  the  support  cortractH^  Prioe? 

Is  the  acquisition  sole  soiree? 

Is  OOTS  use  part  of  the  decision  criteria  fbra/\ad? 
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development  of  OOTSbased  systems?  X 
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Is  use  of  an  IPT  structure  identified? 
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Is  use  ofoormnenda I  practices  identified  in  the  SAIVP? 
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Table  1.  Key  Factor  Identification 
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answer  will  be  given  if  the  acquisition  plan  provides  minimum  requirements  and 
objectives  for  either  requirements  or  key  performance  characteristics.  A  no  will  be  given 
if  the  acquisition  plan  does  not  allow  for  flexibility  or  if  the  issue  is  not  addressed. 

2.  Does  the  system  interface  with  other  programs? 

Interfacing  with  other  programs  could  make  some  engineering  requirements  fixed.  These 
requirements  could  take  away  some  of  the  flexibility  engineers  have  in  design  of  the 
system.  A  yes  will  be  given  if  the  acquisition  plan  specifically  addresses  interfacing  with 
other  systems  or  programs.  This  interface  could  be  either  physical  or  non-physical  such 
as  computer  links.  A  no  will  be  given  if  the  acquisition  plan  identifies  the  program  as 
being  stand-alone  or  if  system  interface  is  not  addressed. 

3 .  Is  this  a  joint  program? 

A  joint  program  is  one  that  is  procured  by  more  than  one  branch  of  the  military.  Joint 
programs  have  an  additional  risk  of  needing  to  satisfy  multiple  users.  This  could  lead  to 
increased  oversight,  schedule  delays,  and  cost  increases.  A  yes  will  be  given  if  the 
acquisition  plan  identifies  the  program  as  being  joint.  A  no  will  be  given  if  the  program 
is  identified  as  being  procured  by  only  one  service  or  is  not  identified. 

4.  Does  the  system  need  to  be  certified  before  being  put  into  operation? 

Certification  typically  requires  adhering  to  standards  of  an  outside  organization  (such  as 
the  Federal  Aviation  Administration).  Certification  was  reviewed  to  see  if  adhering  to 
these  standards  causes  positive  or  negative  effects  on  a  system.  A  yes  will  be  given  if  the 
system  certification  is  mentioned  in  the  acquisition  plan.  A  no  will  be  given  if  the  system 
does  not  need  to  be  certified  or  is  certification  is  not  mentioned  in  the  acquisition  plan. 

An  asterisk  will  be  given  if  certification  does  not  apply  to  the  system. 
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5.  Is  systems  certification  done  by  the  military? 

Systems  that  are  developed  by  the  military  may  have  an  advantage  if  they  are  also 
certified  by  the  military.  Systems  that  are  certified  by  another  organization  may  be  at  a 
disadvantage.  Military  certification  was  reviewed  to  determine  if  programs  are  affected 
by  military  or  outside  certification.  A  yes  will  be  given  if  certification  is  done  by  the 
military.  A  no  will  be  given  if  certification  is  done  outside  the  military  or  certification  is 
not  required.  An  asterisk  will  be  given  if  certification  does  not  apply  to  the  system. 

6.  Is  the  R&D  contract  Cost  Plus? 

Research  and  development  contracts  have  additional  technical  risks  that  are  imposed  on 
the  contractor.  One  means  of  mitigating  this  risk  is  to  use  a  Cost  Plus  type  of  contract.  A 
cost  plus  type  contract  would  allow  the  contractor  to  concentrate  more  on  the  technical 
aspects  of  the  research  without  fear  of  incredible  cost  risks.  A  yes  will  be  given  of  the 
R&D  contract  was  Cost  Plus.  A  no  will  be  given  of  the  contract  is  other  than  cost  type 
contract.  An  asterisk  (*)  will  be  given  if  the  type  of  R&D  contract  is  not  addressed  or  if 
there  was  no  R&D  performed. 

7.  Is  the  support  contract  Fixed  Price? 

Support  contracts  are  generally  considered  to  be  of  lower  risk  to  the  contractor. 

Therefore,  support  contracts  are  generally  fixed  price.  Ayes  will  be  given  of  the  support 
contract  is  fixed  price.  A  no  will  be  given  if  the  contract  is  other  than  fixed  price.  If 
system  support  is  not  addressed  in  the  acquisition  plan,  an  asterisk  will  be  given. 

8.  Is  the  acquisition  sole  source? 

Sole  source  contracts  are  awarded  to  a  single  contractor.  Source  selection  activities  are 
avoided.  This  enables  the  government  and  contractor  to  focus  on  performance  of  the 
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contract  instead  of  awarding  the  contract.  A  yes  will  be  given  for  sole  source 
acquisitions.  If  the  contractor  for  the  program  was  selected  competitively,  a  no  will  be 
given. 

9.  Is  COTS  use  part  of  the  decision  criteria  for  award? 

COTS  use,  as  part  of  the  decision  criteria  for  award,  would  ensure  government  and 
contractor  personnel  know  that  COTS  items  will  be  used  in  the  system  or  the  system  as  a 
whole  will  be  a  COTS-item.  A  yes  will  be  given  if  COTS  use  is  stated  as  award  criteria 
or  the  acquisition  plan  states  that  COTS  use  is  encouraged.  A  no  will  be  given  if  the 
acquisition  plan  states  hat  COTS  will  not  be  a  criteria  for  award  or  if  COTS  use  is  not 
addressed  in  the  acquisition  plan. 

10.  Is  the  prime  contractor  required  to  have  experience  in  the  development  of  COTS- 
based  systems ? 

In  developing  COTS-based  systems,  integrating  commercial  items  requires  extensive 
expertise  (Albert  and  Morris,  2000).  Experience  is  a  critical  factor  to  success  of  a  COTS- 
based  system  (Grant,  2000).  A  yes  will  be  given  if  the  acquisition  plan  sates  that  the 
prime  contractor  is  required  to  have  expertise/experience  in  development  of  COTS-based 
systems.  A  no  will  be  given  if  the  expertise/experience  is  not  required  or  is  not 
mentioned.  An  asterisk  will  be  given  if  experience  in  developing  COTS-based  systems 
does  not  apply. 

1 1 .  Is  open-systems  architecture  used? 

System  architecture  must  be  flexible  enough  to  incorporate  new  releases  of  commercial 
items  and  to  remove  obsolete  commercial  items  (Albert  and  Morris,  2000).  Open 
systems  architecture  combines  standard  interfaces  with  modularity  of  components.  This 
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allows  for  the  flexibility  to  incorporate  new  releases  and  remove  obsolete  items.  Ayes 
will  be  given  if  the  acquisition  plan  states  that  open  systems  architecture  was  used.  A  no 
will  be  given  if  open  systems  is  not  used  or  if  open  systems  is  not  addressed. 

12.  Is  a  plan  for  upgrades/obsolescence  included ? 

Most  commercial  items  must  eventually  be  upgraded  (Albert  and  Morris,  2000).  In  order 
to  maintain  vendor  support  or  replace  obsolete  items,  upgrades  must  be  done.  A  yes  will 
be  given  if  the  acquisition  plan  identifies  a  plan  for  upgrades.  A  no  will  be  given  if  there 
is  no  plan  for  upgrades  or  if  a  plan  for  upgrades  is  not  addressed. 

13.  Is  modification  of  COTS  items  unacceptable ? 

Modification  of  commercial  items  can  lead  to  program  failure  (Albert  and  Morris,  2000). 
Even  if  the  modification  is  unavoidable,  program  risk  is  increased.  Modification  of 
commercial  items  makes  the  item  government  unique.  Vendors  may  not  support  the 
item  and  upgrades  of  the  item  may  not  be  compatible  with  the  system.  A  yes  will  be 
given  if  modification  of  COTS  items  is  not  allowed.  A  no  will  be  given  if  modification  is 
allowed  or  if  modification  is  not  addressed  in  the  acquisition  plan. 

14.  Will  the  military  retain  data  rights  to  the  item? 

Licenses  and  data  rights  can  define  the  relationship  with  the  vendor  (Albert  and  Morris, 
2000).  If  the  government  will  retain  any  or  all  of  the  data  rights,  a  yes  will  be  given.  If 
the  government  will  not  retain  data  rights,  or  data  rights  are  not  addressed,  a  no  will  be 
given. 

15.  Will  the  prime  contractor  support  the  system  throughout  the  entire  life  cycle? 
Having  to  provide  support  for  the  system  after  development  could  provide 
encouragement  to  the  prime  contractor  to  engineer  the  system  for  ease  of  maintenance. 
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Failure  to  engineer  the  system  for  life  cycle  support  could  result  in  a  system  that  cannot 
be  maintained  as  vendors  drop  support  for  obsolete  items  (Albert  and  Morris,  2000).  A 
yes  will  be  given  if  the  acquisition  plan  identifies  that  the  prime  contractor  will  provide 
support  for  the  system.  A  no  will  be  given  of  the  prime  contractor  is  not  required  to 
provide  support  or  if  support  is  not  addressed. 

16.  Is  a  warranty  from  the  prime  contractor  included ? 

A  system  may  have  hidden  costs  due  to  warranties,  especially  of  the  commercial 
warranty  does  not  suit  your  needs  (Carney  and  Obemdorf,  1997).  On  the  other  hand, 
warranties  with  cost  savings  co-sharing  allow  for  reduction  in  total  ownership  costs 
(Grant,  2000).  As  yes  will  be  given  if  a  warranty  for  any  or  all  of  the  system  is  included 
in  the  acquisition  plan.  A  no  will  be  given  if  warranties  are  not  provided  or  not 
addressed. 

1 7.  Is  testing  on  a  system  test  bed  required  before  upgrades  are  included  in  the 
system ? 

Carney  and  Obemdorf  recommend  as  one  of  their  commandments  of  COTS,  “Understand 
the  impact  of  COTS  products  on  the  testing  process”  (Camey  and  Obemdorf,  1997). 
System  level  testing  of  all  COTS  items  needs  to  be  accomplished  to  avoid  disaster. 

Albert  and  Morris  support  this  by  stating,  “A  test  bed  is  an  excellent  mechanism  for 
gaining  insight  into  the  design  and  behavior  of  a  commercial  item”  (Albert  and  Morris, 
2000).  Ayes  will  be  given  for  systems  that  identify  a  requirement  for  test  beds  to  be 
used.  Ayes  will  also  be  given  for  those  systems  that  test  items  on  an  actual  end  item  (i.e. 
one  aircraft)  before  inclusion  into  all  systems  (i.e.  entire  fleet  of  those  aircraft).  A  no  will 
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be  given  if  testing  on  a  system  test  bed  is  not  required  or  is  not  addressed  in  the 
acquisition  plan. 

18.  Is  the  contractor  included  in  government  IPTs? 

Including  the  contractor  in  government  integrated  product  teams  (IPT)  allows  for  the 
contractor  to  be  involved  in  tradeoff  discussions  when  possible  (Grant,  2000).  For 
acquisition  plans  that  identify  contractors  as  part  of  some  or  all  of  the  IPTs,  a  yes  will  be 
given.  If  the  acquisition  plan  states  that  the  contractor  is  not  allowed  to  participate  in 
government  IPTs,  the  issue  is  not  addressed,  or  IPTs  are  not  used,  a  no  will  be  given. 

19.  Is  an  IPT  structure  used? 

Ayes  will  be  given  if  IPTs  are  used  in  the  program.  A  no  will  be  given  if  IPTs  are  not 
used  or  not  addressed. 

20.  Is  use  of  commercial  practices  identified  in  the  acquisition  plan ? 

Use  of  COTS  items  requires  use  of  commercial  practices  that  are  required  with  the 
commercial  item  (Albert  and  Morris,  2000).  When  purchasing  COTS-based  systems,  the 
DoD  must  be  prepared  to  operate  in  a  more  commercial  manner  (Brownsword  and  Place, 
2000).  If  use  of  commercial  practices  is  identified  in  the  acquisition  plan,  a  yes  will  be 
given.  If  commercial  practices  are  not  used,  or  the  subject  is  not  addressed,  a  no  will  be 
given. 

21.  Zs  CAIV/tradeoff  analysis  addressed ? 

Tradeoff  analysis  is  essential  to  a  successful  COTS-based  system  (Grant,  2000).  Ayes 
will  be  given  if  CAIV  or  tradeoff  analysis  is  used.  A  no  will  be  given  if  CAIV  or  tradeoff 
analysis  are  not  used  or  are  not  addressed. 
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22.  Is  TOC  used  in  tracking  costs ? 

Commercial  and  DoD  programs  frequently  underestimate  sustainment  costs  associated 
with  COTS-based  systems.  Therefore,  program  decisions  should  reflect  total  ownership 
costs  (TOC)  (Albert  and  Morris,  2000).  Using  TOC  in  a  COTS-based  system  promotes 
reduced  costs  over  the  life  cycle  of  a  weapons  system  (Grant,  2000).  Ayes  will  be  given 
if  TOC  is  used.  A  no  will  be  given  if  TOC  is  not  used  or  is  not  addressed. 

Critical  Items.  The  critical  items  were  chosen  from  the  investigative  questions. 
These  questions  were  determined  as  critical  to  the  success  of  a  COTS-based  system 
because  the  underlying  concepts  were  identified  in  more  than  one  source  as 
recommended  for  COTS-based  systems.  The  critical  questions  were  not  only  reviewed  to 
see  if  they  were  exclusive  to  successful  systems,  but  also  reviewed  to  see  if  a  certain 
number  of  critical  items  need  to  be  present  for  the  system  to  be  successful. 

Qualitative  Follow  up.  Once  key  success  factors  and  critical  items  were 
determined,  interview  questions  were  developed.  The  interview  questions  were  used  to 
determine  how  the  key  success  factors  affected  program  success.  These  questions  were 
asked  to  key  acquisition  personnel  from  the  applicable  program. 

Summary 

This  chapter  provided  the  information  necessary  to  develop  the  point  that  more 
guidance  is  needed  in  developing  an  acquisition  plan  for  COTS-based  systems.  COTS- 
based  systems  provide  the  DOD  with  a  means  of  incorporating  commercial  items  into 
military  systems.  These  systems  can  provide  benefits  such  as  lower  development  costs 
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and  rapid  technology  insertion.  COTS-based  systems  also  allow  the  DOD  to  give  the 
warfighter  the  military  advantage.  However,  there  are  some  problems  with  using  COTS- 
based  systems.  Inflexible  requirements  and  technology  upgrades  can  lead  to  higher  costs 
than  initially  planned.  Failure  to  adequately  budget  for  appropriate  life-cycle  costs  also 
leads  to  problems  with  COTS-based  systems.  These  problems  are  attributable  to  risks 
associated  with  COTS-based  systems  from  software/hardware  upgrades,  product  quality, 
military  security,  and  funding.  These  risks  can  be  reduced  through  a  change  in  thinking, 
use  of  commercial  practices,  better  configuration  management,  and  choosing  the  correct 
vendor.  However,  risk  reduction  is  not  enough  to  ensure  successful  implementation  of 
COTS-based  systems.  An  acquisition  plan  tailored  to  a  COTS-based  system  is  needed  to 
ensure  the  system  can  be  highly  successful.  While  there  is  some  information  available 
for  developing  an  acquisition  plan  for  COTS-based  systems,  acquisition  professionals 
need  more  guidance  in  specific  areas. 

Chapter  3,  Methodology,  will  explore  the  research  methods  used  to  develop  the 
guidance  acquisition  professionals  need  with  regard  to  COTS  acquisition  strategy.  It  will 
relate  how  programs  will  be  studied  and  how  the  acquisition  plans  will  be  studied. 
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III.  Methodology 


Overview 

The  chapter  begins  by  providing  the  research  questions  developed  for  the  study  of 
acquisition  plans.  The  rationale  is  provided  for  studying  the  cases  analyzed  by  the 
Software  Advisory  Board  (SAB).  Next,  the  chapter  explores  the  research  methods 
available  for  conducting  the  analysis  in  Chapter  4.  Case  study  research  is  presented  as 
the  appropriate  research  method.  The  process  of  data  analysis  is  then  reviewed.  This 
includes  stating  how  the  analysis  in  chapter  4  was  conducted.  The  chapter  ends  by 
providing  criteria  for  evaluating  the  quality  of  research. 

Research  Questions 

The  objective  of  this  research  is  to  identify  some  critical  success  factors  that  need 
to  be  included  in  the  acquisition  plan  of  a  COTS-based  system  for  the  program  to  be 
considered  highly  successful.  Determination  of  ‘highly  successful’  was  made  by  the 
SAB,  and  their  criteria  for  such  a  determination  are  covered  in  the  next  section.  There 
were  no  anticipated  results  prior  to  conducting  the  research  and  data  collection. 
Acquisition  plans  are  complex  documents  that  convey  the  acquisition  strategy  of  a 
program.  Acquisition  plans  contain  information  about  the  program  content,  acquisition 
strategy,  engineering  and  technical  approach,  support  strategy,  test  strategy,  management 
strategy,  and  financial  management  of  a  program.  The  information  in  an  acquisition  plan 
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is  at  the  strategic  level.  A  detailed  analysis  of  each  specific  area  is  usually  included  in 
another  plan,  such  as  the  Test  and  Evaluation  Master  Plan  (TEMP). 

The  strategic  level  of  the  acquisition  plan  provides  an  upper-level  view  of  the 
entire  program,  rather  than  analysis  of  a  specific  area.  For  this  reason,  the  scope  of  this 
research  was  limited  to  the  strategic  level  view  through  the  posing  of  three  research 
questions.  The  three  research  questions  are: 

1 .  Is  there  a  relationship  between  key  factors  of  an  acquisition  plan  and  highly 
successful  programs? 

2.  How  do  the  key  success  factors  affect  the  success  of  the  program? 

3.  How  many  critical  items  need  to  be  present  in  the  acquisition  plan  for  the 
program  to  be  rated  highly  successful  by  the  SAB? 

Critical  items  are  those  that  are  recommended  for  inclusion  in  COTS-based  systems  by 

more  than  one  source. 

Scientific  Advisory  Board 

The  cases  studied  in  this  thesis  were  all  part  of  a  previous  Air  Force  Scientific 
Advisory  Board  study  done  in  April  2000.  Representatives  from  both  government  and 
industry  were  included  on  the  SAB.  The  purpose  of  the  SAB  was  to  “develop  a  checklist 
of  actions  that  need  to  take  place  in  order  to  ensure  the  integration  of  COTS  into  Air 
Force  systems  results  in  products  that  perform  as  advertised  initially  and  through 
subsequent  upgrades,  are  affordable  through  their  life  cycle,  are  safe,  are  not  made 
obsolete  by  a  vanishing  or  changing  industrial  base”  (Grant,  2000:Intro).  Determination 
of  ‘highly  successful’  was  made  based  on  these  factors.  The  highly  successful  programs 
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were  selected  to  “represent  the  best  program  attributes  by  both  government  and  industry 
officials”  (Grant,  2000:18).  The  cases  identified  by  the  SAB  were  used  in  this  study 
because  the  determination  of  ‘highly  successful’  had  already  been  made.  Both  the 
‘highly  successful’  cases  and  the  other  than  highly  successful  cases  were  taken  from  the 
SAB  study.  Using  these  cases  allowed  the  researcher  to  remain  unbiased  at  determining 
the  success  of  the  program. 


Research  Method  Selection 


The  table  below  provides  a  method  to  determine  which  research  method  to  use. 


Strategy 

Form  of  Research 
Question 

Requires  Control 
Over  Behavioral 
Events? 

Focuses  on 

Contemporary 

Events? 

Experiment 

how,  why 

yes 

no 

Survey 

who,  what,*  where, 
how  many,  how  much 

no 

yes 

Archival  Analysis 
(e.g.,  economic 
study) 

who,  what,*  where, 
how  many,  how  much 

no 

yes/no 

History 

how,  why 

no 

no 

Case  Study 

how,  why 

no 

yes 

•  “What”  questions,  when  asked  as  a  part  of  an  exploratory  study,  pertain  to  all  five 


strategies  (Yin,  1994:33) 


Table  2.  Relevant  Situations  for  Different  Research  Strategies 
Three  aspects  of  the  research  are  analyzed  to  determine  which  strategy  is  appropriate. 
The  first  aspect  of  research  reviewed  in  developing  a  research  strategy  is  the  form  of  the 
research  questions.  The  form  of  the  research  questions  in  this  study  are  both  how  and 
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what.  This  eliminates  the  archival  analysis  and  survey  strategies.  The  next  question  to 
answer  is  "Does  the  research  require  control  over  behavioral  events?"  The  research 
questions  in  this  effort  do  not  require  control  over  behavioral  events;  therefore,  the 
experiment  strategy  was  not  used.  The  final  question  to  answer  in  determining  research 
strategy  is  "Does  the  research  focus  on  contemporary  events?"  The  research  questions  do 
focus  on  contemporary  events;  thus  the  history  strategy  was  eliminated.  By  focusing  on 
the  questions  posed  by  Yin,  a  case  study  strategy  was  the  only  appropriate  strategy  for  the 
research  questions.  Therefore,  this  study  used  the  case  study  strategy  to  perform  the 
research. 

Qualitative  Research 

According  to  Strauss  and  Corbin,  qualitative  research  can  be  reported  in  one  of 
three  different  ways.  In  the  first  category  of  reporting  data,  researchers  gather  and  report 
data  without  any  bias  from  the  researcher.  In  the  second  category  of  data  reporting, 
researchers  provide  an  accurate  description  of  the  data.  Since  the  data  gathered  is  usually 
large,  it  needs  to  be  presented  in  a  useful  manner.  In  the  third  category,  researchers  use 
qualitative  research  to  build  theories.  They  believe  that  theories  represent  the  most 
systematic  ways  to  gain  knowledge  (Strauss  and  Corbin,  1990).  This  thesis  tries  to  both 
present  an  ‘accurate  description’  of  acquisition  plans  to  be  used  in  COTS-based  systems 
and  provide  a  theories  that  can  be  used  to  gain  knowledge. 
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Case  Study 


As  defined  by  Yin,  a  case  study  is  an  empirical  inquiry  that  investigates  a 
contemporary  phenomenon  within  its  real-life  context  when  the  boundaries  between 
phenomenon  and  context  are  not  clearly  evident  and  in  which  multiple  sources  of 
evidence  are  used  (Yin,  1994).  Research  into  contracting  techniques  for  COTS  based 
systems  clearly  fits  this  definition.  ‘Real-life’  was  provided  by  the  different  systems  that 
were  studied.  The  systems  studied  supplied  pertinent  information  to  the  phenomenon  of 
COTS-based  systems.  Finally,  multiple  sources  of  evidence  (different  programs)  were 
used.  Therefore,  this  study  fits  the  definition  of  a  case  study. 

Types  of  Case  Studies.  The  type  of  case  study  to  be  used  is  an  embedded, 
multiple  case  design.  An  embedded  design  is  one  in  which  multiple  units  of  analysis  are 
used.  The  key  factors  derived  from  the  acquisition  plans  are  the  multiple  units  being 
examined  in  this  study.  Since  there  are  multiple  units  of  analysis,  this  research  follows 
an  embedded  design.  Since  several  COTS-based  systems  rated  highly  successful  and 
several  not  rated  highly  successful  were  studied,  this  is  also  a  multiple  case  design  (Yin, 
1994).  This  multiple  case,  embedded  research  design  study  was  used  to  gather  data  about 
COTS-based  systems  from  their  acquisition  plans. 

Sources  of  Evidence.  According  to  Yin,  there  are  six  sources  of  evidence: 
documentation,  archival  record,  interviews,  direct  observations,  participant  observations, 
and  physical  artifacts  (Yin,  1994).  The  sources  of  evidence  used  in  this  effort  were 
documentation  (in  the  form  of  acquisition  plans)  and  interviews.  The  acquisition  plans 
from  various  programs  were  used  to  gather  data  for  the  three  research  questions.  In 
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addition  to  the  acquisition  plans,  interviews  were  used  to  provide  information  for 
research  question  three. 

Data  Analysis 

Theory.  Yin  provides  four  techniques  for  analyzing  data  gathered  in  a  case  study: 
pattern-matching,  explanation-building,  time-series  analysis,  and  program  logic  models. 
This  study  used  explanation-building  to  study  the  data.  Explanation-building  involves 
analyzing  the  case  study  by  building  an  explanation  about  the  case.  Explanation-building 
was  used  for  research  question  three.  Yin  further  states  that  analyzing  embedded  units  is 
a  lesser  mode  of  analysis  that  can  be  used  with  explanation-building.  The  embedded 
units  of  analysis  -  the  factors  -  are  studied  first  within  each  case  and  then  across  cases 
(Yin,  1994).  Analyzing  embedded  units  was  used  in  examining  the  three  research 
questions. 

Analysis  Coding.  Three  types  of  analysis  coding  were  used  during  different  parts 
of  the  evaluation  (Strauss  and  Corbin,  1990).  These  coding  types  are  open  coding,  axial 
coding,  and  selective  coding.  Open  coding  is  used  to  obtain  and  document  data  from 
each  of  the  cases.  This  was  done  by  reviewing  the  acquisition  plans  for  key  factors. 

Axial  coding  is  used  to  detect  emerging  phenomena  across  the  cases.  This  was  done  in 
determining  both  the  key  success  factors  and  the  critical  factors  among  the  acquisition 
plans.  Selective  coding  is  used  in  maturing  a  model  to  explain  the  phenomena.  Selective 
coding  was  used  when  gathering  responses  to  the  interview  questions  and  in  developing 
the  ultimate  assertions  I  make  in  Chapter  5. 
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In  performing  the  analysis  of  the  factors,  the  researcher  answered  the  questions 
based  on  the  acquisition  plan  for  that  program.  In  order  to  achieve  increased  validity  and 
reliability  in  this  analysis,  outside  sources  were  also  used  in  factor  analysis.  Acquisition 
plans  from  one  highly  successful  and  one  other  than  highly  successful  program  were 
randomly  selected  for  analysis  by  an  outside  source.  Since  the  answers  provided  by  the 
researcher  matched  the  answers  provided  by  the  outside  source,  validity  and  reliability  of 
the  research  was  increased. 

Analysis  of  Critical  Factors.  Ten  of  the  factors  were  considered  to  be 
critical  factors  (questions  previously  italicized).  They  were  identified  as  critical  factors 
because  underlying  concepts  behind  these  questions  appeared  in  more  than  one  source 
recommending  them  for  use  in  COTS-based  systems.  These  investigative  questions  were 
also  posed  as  yes/no,  but  a  yes  also  meant  that  this  was  a  positive  aspect  for  the  system  to 
have. 

The  programs  were  looked  at  individually  to  determine  how  many  of  the  critical 
factors  were  included  in  that  program.  An  average  was  determined  for  the  number  of 
critical  factors  contained  in  the  acquisition  plans  for  both  highly  successful  and  other  than 
highly  successful  programs.  The  averages  were  then  compared  to  determine  if,  on 
average,  the  highly  successful  programs  contain  more  critical  items  than  the  other  than 
successful  programs.  This  analysis  attempted  to  determined  if  a  certain  number  of 
critical  factors  were  needed  to  be  present  for  a  system  to  be  rated  highly  successful, 
notwithstanding  which  ones  were  present. 

Further  analysis  was  performed  on  the  critical  factors.  The  critical  factors  were 
reviewed  to  determine  which  ones  were  contained  in  all  of  the  successful  programs.  This 
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analysis  was  done  to  see  which  critical  factors  could  be  integral  to  the  success  of  all 
programs. 

Qualitative  Follow-up.  The  qualitative  analysis  needed  for  research  question 
number  3  was  done  by  interview.  The  program  manager,  deputy  program  manger,  or  a 
contracting  officer  from  each  program  was  interviewed  to  see  why  he  or  she  felt  the 
identified  item  led  to  success  of  their  program.  The  interview  question  for  each  aspect 
was  "How  did  (factor)  enabled  the  program  to  be  highly  successful?"  Programs  that  did 
not  include  the  factor  in  the  acquisition  plan  were  asked  if  the  factor  was,  in  fact,  used, 
notwithstanding  its  absence  from  the  acquisition  plan.  If  it  was  used,  they  were  also 
asked  why  it  was  not  included  in  the  acquisition  plan.  Interview  questions  are  presented 
in  Appendix  A.  Explanations  were  built  by  comparing  the  responses  to  the  interviews 
from  the  different  programs.  By  comparing  responses  across  multiple  cases  an 
explanation  was  constructed. 

Criteria  for  Evaluating  Research  Quality 

In  order  to  ensure  the  research  presented  is  of  a  high  quality,  the  research  was 
designed  with  Yin’s  four  tests  in  mind.  Yin  developed  four  tests  applicable  to  case 
studies  to  ensure  research  quality:  construct  validity,  internal  validity,  external  validity, 
and  reliability  (Yin,  1994). 

Construct  Validity.  Construct  validity  relates  to  establishing  the  correct  measure 
for  the  concepts  being  studied.  According  to  Yin,  construct  validity  can  be  achieved  by 
using  multiple  sources  of  evidence.  One  way  of  using  multiple  sources  of  evidence  is 
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data  triangulation  (Yin,  1994).  In  order  to  ensure  construct  validity,  multiple  sources  of 
evidence  were  used  in  this  case  study.  Review  of  current  literature  enabled  identification 
of  the  key  factors.  The  acquisition  plans  were  studied  to  review  the  key  factors. 
Interviews  were  also  accomplished  in  light  of  the  data  gathered  from  the  literature  and 
acquisition  plans.  This  provided  triangulation  of  data  to  ensure  construct  validity  in  the 
research. 

Internal  Validity.  Internal  validity  relates  to  establishing  a  causal  relationship. 
Pattern  matching  is  one  of  the  most  desirable  strategies  in  performing  a  case  study  (Yin, 
1994).  In  this  study,  performing  the  numerical  coding  in  the  data  analysis  and  then 
performing  interviews  in  the  explanation  building  contributed  to  internal  validity. 

Factors  were  only  studied  in  depth  after  the  need  was  determined  by  the  initial  data 
analysis. 

External  Validity.  External  validity  is  the  process  of  establishing  a  population  of 
which  the  results  of  the  study  can  be  applied.  Using  replication  data  in  multiple  case 
studies  is  one  means  of  establishing  external  validity  (Yin,  1994).  External  validity  was 
achieved  in  this  study  by  analyzing  multiple  cases.  All  five  of  the  highly  successful 
programs  and  five  of  the  programs  not  rated  highly  successful  identified  by  the  SAB  were 
used  to  analyze  the  factors. 

Reliability.  Reliability  in  a  case  study  deals  with  the  ability  to  repeat  the  findings 
with  the  same  results.  Reliability  is  enhanced  by  using  a  case  study  protocol  (Yin,  1994). 
A  case  study  protocol  was  used  in  this  research.  Data  was  coded  first  by  the  researcher. 
Then  others  replicated  coding  20%  of  the  data.  The  results  of  the  researcher  and  the 
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others  were  the  same.  Additionally,  the  interview  protocol  used  (Appendix  A)  ensured 
that  the  results  of  the  interviews  were  reliable. 

Summary 

To  gather  contracting  data  from  existing  COTS-based  systems,  an  embedded 
multiple  case  study  design  was  performed.  A  pattern  matching  technique  was  used  that 
employed  first  within-case  and  then  across-case  analysis.  The  pattern-matching  analysis 
provided  enough  information  to  further  perform  a  qualitative  analysis  on  each  identified 
key  success  factor.  The  result  of  this  study  provided  enough  information  to  develop  a 
theory  that  will  aid  acquisition  professionals  in  the  development  of  acquisition  plans  for 
use  in  COTS-based  systems. 
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IV.  General  Results  and  Analysis 


Overview 

This  chapter  presents  the  results  of  the  study  outlined  in  chapter  3.  First,  the  key 
factors  are  analyzed  in  identifying  the  possible  key  success  factors.  Since  they  were 
found  to  be  possible  key  success  factors,  a  review  on  total  ownership  cost  (TOC)  and  cost 
as  an  independent  variable  (CAIV)  is  presented.  Next,  the  critical  factors  are  reviewed 
across  the  cases  to  determine  if  a  certain  number  of  critical  items  have  an  effect  on 
program  success.  Finally,  qualitative  analysis  is  performed  to  determine  why  certain 
factors  lead  to  program  success. 

Key  Factor  Analysis 

This  part  of  the  research  was  accomplished  to  identify  key  success  factors  in  the 
acquisition  plans  of  COTS-based  systems.  Key  success  factors  are  elements  of  the 
acquisition  plan  that  correlate  to  a  COTS-based  system  being  successful.  In  order  to  do 
this,  the  data  was  coded  from  reviewing  the  acquisition  plans  of  each  program  for  each 
factor.  Table  3  summarizes  the  results  of  the  research.  The  table  shows  the  questions  on 
the  left  side  with  the  applicable  programs  across  the  top.  Critical  questions,  as  defined  in 
Chapter  3,  are  italicized.  Programs  not  rated  highly  successful  by  the  SAB  are  labeled 
cases  A  through  E.  The  programs  rated  highly  successful  are  labeled  cases  F  through  J. 
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The  results  of  the  analysis  suggest  that  the  only  key  success  factors  are  TOC  and  CAIV. 
The  following  is  an  analysis  of  each  question. 

1 .  Are  the  requirements  flexible ? 

Among  the  acquisition  plans  from  the  rated  highly  successful  systems,  four  programs 
allowed  for  flexibility  of  requirements.  Three  of  the  not  rated  highly  successful  programs 
identified  flexibility  of  requirements  in  their  acquisition  plans.  These  cases  either 
specifically  stated  the  requirements  were  flexible  or  identified  requirements  in  terms  of 
minimum  objectives  and  goals.  The  one  rated  highly  successful  case  that  was  coded  no, 
case  G,  did  not  mention  flexibility  in  the  acquisition  plan.  However,  the  procurement 
contracting  officer  stated  “Since  CAIV  was  included  in  the  acquisition  plan,  flexible 
requirements  were  a  given.”  Identifying  flexible  requirements  would  have  been 
redundant.  One  not  rated  highly  successful  case,  case  C,  did  not  mention  flexibility 
either.  Case  D,  while  coded  no,  did  mention  trade  off  analysis  on  the  basis  of  cost, 
schedule,  risk,  and  performance.  Overall,  four  rated  highly  successful  and  three  not  rated 
highly  successful  systems  included  flexibility  of  requirements  in  the  acquisition  plans. 
The  difference  of  1  between  the  two  groups  suggests  that  flexibility  of  requirements  is 
not  a  key  success  factor. 

2.  Does  the  system  interface  with  other  programs? 

All  systems,  except  case  G,  had  acquisition  plans  that  addressed  interface  with  another 
program.  Most  systems  interfaced  electronically  with  other  systems,  such  as  aircraft 
systems  communicating  with  other  aircraft.  Cases  D,  H,  and  I  specifically  mention 
electronic  and  physical  interface  with  another  systems.  Nine  of  the  10  programs  address 
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Table  3.  Key  Factor  Analysis 


interface  issues.  The  difference  of  1  between  the  two  groups  suggests  that  interfacing 
with  other  programs  is  not  a  key  success  factor. 

3 .  Is  this  a  j oint  program? 

Cases  A,  H,  and  I  are  all  joint  programs.  The  seven  other  cases  are  not  joint.  Overall, 
two  rated  highly  successful  programs  and  one  not  rated  highly  successful  program  are 
joint  programs.  The  difference  of  1  between  the  two  groups  suggests  that  a  program 
being  joint  is  not  a  key  success  factor. 

4.  Does  the  system  need  to  be  certified  before  being  put  into  operation? 

Four  programs  did  not  identify  certification  in  their  acquisition  plans  -  cases  B,  D,  G, 
and  H.  In  case  I,  certification  did  not  apply.  All  other  cases  identified  a  certification 
requirement.  Overall,  the  acquisition  plans  of  2  rated  highly  successful  programs  and 
three  not  rated  highly  successful  programs  identified  a  need  for  certification.  The 
difference  of  1  between  the  two  groups  suggests  that  certification  is  not  a  key  success 
factor. 

5.  Is  systems  certification  done  by  the  military? 

All  systems  that  required  certification  needed  to  be  certified  by  the  military.  The 
difference  of  1  between  the  two  groups  suggests  that  military  certification  is  not  a  key 
success  factor. 

6.  Is  the  R&D  contract  type  cost  plus? 

Three  of  acquisition  plans  of  the  rated  highly  successful  programs  identified  a  cost  plus 
type  of  contract  for  R&D.  Of  those,  case  G  used  a  cost  plus  fixed  fee  contract  and  case  J 
used  a  cost-plus  award  fee  contract.  Case  F,  coded  with  an  asterisk,  was  an  overarching 
widely-scoped  Indefinite  Delivery  /  Indefinite  Quantity  (IDIQ)  contract  that  allowed  for 
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flexibility  in  determining  contract  type  on  each  specific  delivery  order.  Case  H  did  not 
have  R&D  performed.  The  acquisition  plans  of  three  of  the  not  rated  highly  successful 
programs  also  identified  a  cost  plus  type  of  contract  for  R&D.  Cases  B,  D  and  E  all  used 
cost  plus  award  fee  contracts  for  R&D.  Case  A  used  a  fixed  price  incentive  fee  contract. 
Case  C,  coded  with  an  asterisk,  was  also  an  IDIQ  type  contract.  The  difference  of  0 
between  the  two  groups  suggests  that  using  a  cost  plus  contract  for  R&D  is  not  a  key 
success  factor. 

7.  Is  the  support  contract  Fixed  Price? 

Of  the  rated  highly  successful  programs,  the  acquisition  plans  for  cases  G  and  J  did  not 
address  type  of  contract  for  system  support  in  the  SAMP.  Case  F,  coded  with  an  asterisk, 
used  an  IDIQ  format.  Cases  H  and  I  both  used  a  fixed  price  type  of  contract  for  support. 
Of  the  not  rated  highly  successful  programs,  case  C  used  an  IDIQ  format.  All  other 
programs  used  fixed  price  contract  for  system  support.  The  difference  of  2  between  the 
two  groups  suggests  that  using  a  fixed  price  contract  for  support  is  not  a  key  success 
factor. 

8.  Is  the  acquisition  sole  source? 

Of  the  rated  highly  successful  programs,  contracts  for  cases  F  and  G  were  awarded  sole 
source.  For  cases  H,  I,  and  J  the  contractor  was  selected  on  a  competitive  basis.  Of  the 
not  rated  highly  successful  programs,  all  but  case  C  were  awarded  on  a  competitive  basis. 
The  difference  of  1  between  the  two  groups  suggests  that  sole  sourcing  is  not  a  key 
success  factor. 
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9.  Is  COTS-use  part  of  the  decision  criteria  for  award? 

Of  the  rated  highly  successful  programs,  the  acquisition  plans  of  three  cases  required 
COTS  use  in  the  decision  criteria  for  award.  Case  H  states  that  the  program  “will  acquire 
a  COTS  application  to  meet  the  required  functionality.”  The  acquisition  plan  for  case  G 
states  “use  of  COTS. .  .to  meet  performance  specification  requirements  is  encouraged.” 
Case  J  included  a  similar  statement.  Case  F  mentions  COTS  use,  but  does  not  specify  it 
as  decision  criteria  in  any  way.  Of  the  programs  not  rated  highly  successful,  three 
required  COTS  use  in  the  decision  criteria  for  award.  The  acquisition  plan  for  case  A 
stated  the  need  to  procure  commercial  items  wherever  possible.  Case  B  contained  a 
similar  statement.  The  difference  of  0  between  the  two  groups  suggests  that  COTS-use 
as  a  decision  criteria  for  award  is  not  a  key  success  factor. 

10.  Is  the  prime  contractor  required  to  have  experience  in  the  development  of  COTS- 
based  systems? 

The  acquisition  plans  of  one  program  from  each  category  required  the  contractor  to  have 
experience  in  the  development  of  COTS-based  systems.  The  difference  of  0  between  the 
two  groups  suggests  that  COTS  development  experience  is  not  a  key  success  factor. 

1 1 .  Is  open-systems  architecture  used? 

Of  the  rated  highly  successful  programs,  the  acquisition  plans  of  three  identified  use  of  an 
open  systems  architecture.  The  acquisition  plan  for  case  G  provided  for  open-systems 
use  by  stating  that  the  program  would  use  an  open  systems  architecture  by  emphasizing 
COTS  and  other  non-developmental  items  in  hardware/software  introduction.  The 
acquisition  plan  for  case  J  provided  several  paragraphs  on  the  use  of  open  systems  and 
open  systems  design.  The  acquisition  plan  for  case  H  stated  the  application  must  be 
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capable  of  operating  in  an  open  system.  Two  of  the  not  rated  highly  successful  programs 
identified  use  of  an  open  systems  architecture.  The  acquisition  plan  for  case  D  stated  “  a 
development  methodology  will  be  implemented  that  provides  for  installation  and 
validation  of  new  functions  in  a  new  distributed  open  systems  architecture.”  The 
difference  of  1  between  the  two  groups  suggests  that  use  of  open  system  architecture  is 
not  a  key  success  factor. 

12.  Is  a  plan  for  upgrades/obsolescence  included ? 

All  of  the  rated  highly  successful  programs  identified  a  plan  for  upgrade/obsolescence  in 
their  acquisition  plans.  The  contract  type  (IDIQ)  and  contract  duration  of  18  years  for 
case  F  allows  for  upgrades  to  the  system.  Case  J  included  plans  for  upgrades  in  the 
section  on  open  systems  design.  Four  of  the  not  rated  highly  successful  programs  include 
a  plan  for  upgrades/obsolescence.  Case  B  requires  the  prime  contractor  to  develop  a 
capability  to  provide  updates.  The  difference  of  1  between  the  two  groups  suggests  that 
including  a  plan  for  upgrades/obsolescence  is  not  a  key  success  factor. 

13.  Is  modification  of  COTS  items  unacceptable ? 

One  of  the  acquisition  plans  of  the  rated  highly  successful  programs,  case  H,  did  not 
allow  for  modification  of  the  COTS  items.  This  acquisition  plan  stated  that 
enhancements  would  only  be  done  as  the  developer  released  upgrades  to  the  program. 

The  acquisition  plan  for  case  I  did  not  address  the  issue.  The  acquisition  plans  of  all 
other  programs  did  not  restrict  the  modification  of  COTS  items.  The  difference  of  1 
between  the  two  groups  suggests  that  restriction  of  modification  is  not  a  key  success 
factor. 
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14.  Will  the  military  retain  data  rights  to  the  item? 

In  the  rated  highly  successful  programs,  two  acquisition  plans  stated  that  data  rights 
would  be  retained  by  the  military.  The  acquisition  plan  for  one  rated  highly  successful 
program  did  not  address  data  rights.  In  the  not  rated  highly  successful  programs,  again 
two  acquisition  plans  stated  the  data  rights  would  be  retained  by  the  military.  Case  C 
stated  that  limited  data  rights  would  be  acquired.  All  other  acquisition  plans  stated  that 
data  rights  would  not  be  retained  by  the  military.  The  difference  of  0  between  the  two 
groups  suggests  that  retention  of  data  rights  is  not  a  key  success  factor. 

1 5.  Will  the  prime  contractor  support  the  system  throughout  the  entire  life  cycle? 

All  programs  except  case  J  required  prime  contractor  support  for  the  system  throughout 
the  entire  life  cycle.  The  acquisition  plan  for  case  J  had  a  support  plan  but  did  not  have  a 
contractor  selected.  The  difference  of  1  between  the  two  groups  suggests  that  prime 
contractor  support  is  not  a  key  success  factor. 

16.  Is  a  warranty  from  the  prime  contractor  included ? 

All  of  the  acquisition  plans  of  the  rated  highly  successful  programs  provided  a  warranty 
from  the  prime  contractor.  Case  F  sought  warranty  protection  for  new  systems  with 
enforcement  of  COTS  warranties.  Case  G  required  the  use  of  commercial  warranties. 
Three  of  the  acquisition  plans  for  the  not  rated  highly  successful  programs  included 
warranties  from  the  prime  contractor.  Cases  A  and  B  require  the  prime  contractor  to 
warrant  the  system  and  administer  all  vendor  warranties.  The  acquisition  plan  for  case  D, 
while  being  coded  no,  stated  that  warranties  may  be  applicable  to  firm  fixed  price 
modification  only.  The  difference  of  2  between  the  two  groups  suggests  that  prime 
contractor  warranty  is  not  a  key  success  factor. 
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17.  Is  testing  on  a  system  test  bed  required  before  upgrades  are  included  in  the 
system ? 

Three  of  the  acquisition  plans  from  each  category  included  requirements  for  testing  on  a 
systems  test  bed  before  upgrades  are  included  in  the  system.  The  acquisition  plan  for 
case  D  required  a  full  test  lab  with  simulation  while  case  C  required  upgrade  testing  on  a 
test  system.  The  acquisition  plan  for  case  B  did  not  directly  address  testing  for  upgrades, 
but  made  reference  to  the  TEMP.  Case  B  was  coded  no.  Of  the  rated  highly  successful 
programs,  case  F  required  testing  before  installation  in  the  system.  Case  J,  coded  no, 
addressed  a  system  test  bed,  but  did  not  specifically  address  upgrades  being  tested  in  the 
test  bed  The  difference  of  0  between  the  two  groups  suggests  that  testing  before 
upgrades  is  not  a  key  success  factor. 

18.  Is  use  of  Integrated  Product  Teams  (IPTs)  identified? 

Acquisition  plans  for  five  of  the  rated  highly  successful  and  four  of  the  not  rated  highly 
successful  programs  included  use  of  an  IPT  structure.  The  difference  of  1  between  the 
two  groups  suggests  that  use  of  IPTs  is  not  a  key  success  factor. 

19.  Is  the  contractor  included  in  government  IPTs? 

Acquisition  plans  for  five  of  the  rated  highly  successful  and  four  of  the  not  rated  highly 
successful  programs  had  provisions  for  including  the  prime  contractor  in  IPTs.  The 
difference  of  1  between  the  two  groups  suggests  that  including  contractors  in  IPTs  is  not 
a  key  success  factor. 

20.  Is  use  of  commercial  practices  identified  in  the  acquisition  plan? 

Among  the  acquisition  plans  for  the  rated  highly  successful  programs,  four  programs 
identified  use  of  commercial  practices.  The  acquisition  plan  for  case  F  makes  no  mention 
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of  commercial  practices  or  best  practices.  The  acquisition  plan  for  case  G,  coded  yes,  did 
not  mention  commercial  practices,  but  did  state  “contractors  are  encouraged  to  further 
streamline  activities”  and  that  “commercial  warranties  would  be  used.”  The  acquisition 
plan  from  case  J,  coded  yes,  identified  use  of  best  practices  from  acquisition  reform. 
Among  the  acquisition  plans  for  the  not  rated  highly  successful  programs,  four  programs 
identified  use  of  commercial  practices.  The  acquisition  plan  for  case  A  while  calling  for 
use  of  commercial  practices  also  stated  a  task  force  was  formed  to  make  the  acquisition 
more  like  a  commercial  acquisition.  The  acquisition  plans  for  cases  B  and  C  encouraged 
the  use  of  commercial  practices.  The  difference  of  0  between  the  two  groups  suggests 
that  inclusion  of  commercial  practices  in  the  acquisition  plan  is  not  a  key  success  factor. 

21 .  Is  CAIV  analysis  used ? 

All  five  of  the  rated  highly  successful  programs  identified  CAIV  in  their  acquisition 
plans.  Case  H  did  not  identify  CAIV  specifically.  However,  the  program  manager  stated 
that  CAIV  was  considered  in  the  TOC  analysis  that  was  included  in  the  acquisition  plan. 
Therefore,  case  H  was  determined  to  be  a  yes.  Two  of  the  not  rated  highly  successful 
programs  identified  use  of  CAIV.  Three  did  not  use  CAIV.  However,  according  to  the 
acquisition  plan,  case  D  did  use  a  tradeoff  analysis  that  “will  be  conducted  throughout  the 
life  of  the  contract  on  an  as  required  basis  for  cost,  schedule,  risk,  and  performance.” 

This  seemed  to  be  similar  to  trade  space  analysis  done  in  a  CAIV  analysis.  Although  it 
did  seem  similar  to  CAIV,  this  was  classified  as  a  no.  The  difference  of  3  between  the 
two  groups  suggests  that  CAIV  is  a  key  success  factor. 
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22.  Is  TOC  used  in  tracking  costs ? 

All  five  of  the  rated  highly  successful  systems  used  TOC  to  track  costs.  The  acquisition 
plan  for  case  F  had  a  full  paragraph  on  TOC  within  the  program.  The  plan  for  case  G 
identified  TOC  used  at  the  system  level,  which  supported  their  department’s  overall 
goals.  The  plan  for  case  J  imposed  TOC  goals  and  measurements  for  contractors 
achieving  those  goals.  Two  of  the  not  rated  highly  successful  programs  identified  TOC 
use  for  tracking  costs.  Case  A,  coded  yes,  identified  lowest  total  system  life  cycle  cost  as 
a  factor  in  evaluating  proposals.  The  plan  for  case  C,  coded  yes,  identified  a 
“performance  based  business  focus  on  RTOC”.  The  acquisition  plan  for  case  B,  coded 
no,  identified  a  life  cycle  cost  model  in  cost  budgeting,  but  not  in  actual  tracking  of  costs. 
The  plan  for  case  D,  coded  no,  stated  that  by  increasing  overall  reliability  and 
maintenance,  overall  LCC  will  be  reduced,  but  did  not  identify  TOC.  The  acquisition 
plan  for  case  E  identified  TOC  as  being  used.  However,  since  the  acquisition  plan  did 
not  consider  O&M  cost  in  TOC,  the  case  was  coded  no.  The  difference  of  3  between  the 
two  groups  suggests  that  use  of  TOC  is  a  key  success  factor. 

In  this  part  of  the  research,  two  key  success  factors  were  identified.  The  first  key 
success  factor  found  was  including  CAIV  analysis  in  the  acquisition  plan.  Additionally, 
using  TOC  in  tracking  costs  was  identified  as  a  key  success  factor.  These  two  factors  are 
reviewed  in  the  next  section. 
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TOC/CAIV 


In  order  to  understand  Total  Ownership  Cost  (TOC),  as  used  by  the  Air  Force,  the 
terms  must  first  be  defined.  The  Air  Force  views  TOC  in  two  different  ways,  DoD  TOC 
and  Defense  Systems  TOC.  The  Air  Force  Reduction  in  Total  Ownership  Cost 
CAIV/TOC  Guidebook  presents  DoD  TOC  as  .  .the  sum  of  all  financial  resources 
necessary  to  organize,  equip,  train,  sustain,  and  operate  military  forces  sufficient  to  meet 
national  goals...”  (Reduction,  1999:5).  The  guidebook  defines  Defense  System  TOC  as 
Life  Cycle  Costs  (LCC). 

LCC  includes  not  only  acquisition  program  direct  costs,  but  also  indirect  costs 
attributable  to  the  acquisition  program  (i.e.,  costs  that  would  not  occur  if  the 
program  did  not  exist).  For  example,  indirect  costs  would  include  the 
infrastructure  that  plans,  manages,  and  executes  a  program  over  its  full  life  and 
common  support  items  and  systems  (Reduction,  1999:6). 

DoD  TOC  is  a  three  dimensional  concept  consisting  of  Defense  System  Performance  and 

Design,  Resources  to  Operate,  and  Operational/Warfighting  Concepts.  Defense  System 

Performance  and  Design  includes  costs  that  are  a  direct  result  of  weapons  system  design 

such  as  those  considered  in  LCC.  Resources  to  Operate  encompasses  infrastructure  and 

force  structure  costs  not  directly  attributable  to  weapons  systems  such  as  base  operating 

support  (BOS)  or  transportation.  Operational/Warfighter  Concepts  includes  costs  driven 

by  specific  concepts  such  as  the  Air  Expeditionary  Force  (AEF)  concept  (Reduction, 

1999). 

Defense  System  TOC  (LCC)  is  driven  by  requirements  pull  and  technology  push 
as  shown  in  the  Figure  2.  As  the  warfighter  engages  in  new  threats,  their  requirements 
change  which  ‘pulls’  resources.  At  the  same  time,  new  technologies  are  being  developed 
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that  increase  system  performance  and  are  ‘pushed’  into  the  weapons  systems.  To 
maintain  a  system  that  meets  the  new  threats,  incorporates  new  technologies,  and  remains 
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-  Define  Operational  Capabilities  /  Concepts 

-  Satisfy  Operational  Needs 

-  Determine  Weapon  System  Requirements 

-  Exploit  Opportunities 

-  Primary  Influence  /  3400 

-  Primary  Influence  3010  /  3600 

-  Drives  3010  /  3600 

-  Drives  3400 

MUST  MAINTAIN  BALANCE 

-  Requirements  Determination 

-  Technology  Maturation  /  Insertion 

-  Program  Strategy 


Figure  2.  TOC  Drivers  (Reduction,  1999:9) 

cost  effective  a  balance  is  needed  between  operational  capability  and  system  costs 

(Reduction,  1999).  The  means  of  attaining  this  balance  is  through  a  CAIV  analysis. 

Cost-as-an-independent-variable  is  the  primary  strategy  used  in  Defense  Systems 

Performance  and  Design  to  reduce  life  cycle  costs  (defense  systems  TOC).  CAIV  is  used 

in  system  design  to  obtain  the  best  possible  system  with  the  lowest  life  cycle  cost.  CAIV, 

as  defined  in  Air  Force  Instruction  (AFI)  10-601,  is: 

The  process  of  using  better  business  practices,  allowing  “Trade  Space”  for 
industry  to  met  user  requirements,  and  considering  operations  and  maintenance 
costs  early  in  the  requirements  definition  in  order  to  procure  systems  smarter  and 
more  efficiently  (AFI  10-601,  1998:Atch.  1). 

Placing  a  cap  on  systems  cost  is  a  principle  of  CAIV.  Any  additional  funds  needed  must 

be  taken  from  the  program  itself,  not  other  programs  or  force  modernization  efforts. 

Trade  Space  is  another  principle  for  decision  making  when  using  CAIV .  Trade  Space  is 

the  range  of  alternatives  available  to  decision  makers.  Key  performance  parameters 
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(KPP)  are  set  with  thresholds  and  objectives.  Decision  makers  view  the  alternatives  to 
each  KPP  and  try  to  reach  thresholds  without  jeopardizing  objectives  for  other  KPPs  . 
Decisions  are  made  based  upon  impacts  to  cost  (LCC),  schedule,  performance,  and  risk 
(Reduction,  1999).  Decision  makers  try  to  reach  a  decision  that  balances  operational 
requirements  against  life  cycle  costs. 

Critical  Item  Analysis 

This  part  of  the  analysis  focuses  on  the  critical  items  identified  in  Chapter  3.  The 
critical  items  were  all  recommended  for  use  in  COTS-based  systems  by  more  than  one 
source.  The  questions  for  the  critical  items  were  written  so  that  a  yes  answer  was  a 
positive  system  attribute.  The  analysis  was  first  done  to  see  how  many  of  the  critical 
items  were  contained  in  the  acquisition  plans  of  each  program.  The  average  for  the  rated 
highly  successful  programs  was  compared  against  the  average  of  the  not  rated  highly 
successful  systems.  Additionally,  each  critical  factor  was  looked  at  individually  to  see 
which  had  a  unanimous  result  in  either  of  the  categories.  Table  4  shows  the  results  of  this 
analysis. 

The  programs  were  analyzed  to  determine  if  the  number  of  critical  factors 
included  in  the  acquisition  plans  of  the  rated  highly  successful  programs  was  higher  than 
that  of  the  not  rated  highly  successful  programs.  In  this  analysis,  the  rated  highly 
successful  programs  included  an  average  of  7.6  of  the  ten  critical  factors  in  their 
acquisition  plans.  The  not  rated  highly  successful  programs  included  an  average  of  5.2 
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Table  4.  Critical  Item  Analysis 


critical  factors.  This  resulted  in  a  difference  of  2.4  more  critical  factors  included  in  the 
acquisition  plans  of  the  rated  highly  successful  programs  over  the  not  rated  highly 
successful  programs.  Table  4  also  provides  a  subtotal  of  critical  items  before  TOC  and 
CAIV  are  included.  This  subtotal  shows  there  is  minimal  difference  in  the  programs 
before  TOC  and  CAIV  are  included. 

Due  to  the  limited  number  of  samples  (5  each),  this  analysis  was  taken  one  step 
further.  Case  D,  a  not  rated  highly  successful  program,  contained  only  two  of  the  critical 
factors.  No  other  program  came  close  to  having  that  few  critical  factors.  (This  case  was 
ultimately  terminated  at  S  AF/AQ  direction.)  Case  H,  a  rated  highly  successful  program, 
contained  the  most  critical  factors  of  any  program  -  nine.  To  determine  if  these  outlying 
cases  had  an  extreme  effect  on  the  results,  the  analysis  was  conducted  with  these  two 
cases  removed.  With  these  two  extreme  cases  removed  from  the  analysis,  the  rated 
highly  successful  programs  contain  an  average  of  7.25  critical  factors.  The  not  rated 
highly  successful  programs  contain  an  average  of  6  critical  factors.  This  still  provides  a 
difference  of  1 .25  more  critical  factors  for  the  rated  highly  successful  programs. 
Furthermore,  two  of  the  critical  factors  for  case  I  were  identified  as  not  applicable  to  the 
program.  In  the  above  analysis  this  was  factored  with  the  same  weight  as  a  no.  This  was 
seen  as  penalizing  the  highly  successful  programs.  The  analyses  done  without  the 
penalty  results  in  a  difference  of  2.7  before  the  extreme  cases  are  factored  out  and  1.6 
after  the  extreme  cases  are  factored  out.  Therefore,  the  difference  in  the  number  of 
critical  factors  in  the  acquisition  plans  of  rated  highly  successful  programs  versus  the  not 
rated  highly  successful  programs  is  as  high  as  2.7  and  as  low  as  1.25. 
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The  critical  factors  were  also  looked  at  to  determine  which  ones  had  unanimous 
results  in  either  of  the  categories.  Among  the  rated  highly  successful  programs,  five  of 
the  critical  factors  were  included  in  all  of  the  acquisition  plans.  The  critical  factors 
included  in  all  of  the  rated  highly  successful  programs  are:  including  a  plan  for 
upgrades/obsolescence,  a  warranty  from  the  prime  contractor  is  included,  the  contractor 
is  included  in  government  IPTs,  CAIV  analysis  is  used,  and  TOC  is  used  in  tracking 
costs.  CAIV  and  TOC  were  both  also  identified  as  key  success  factors.  Among  the  not 
rated  highly  successful  programs,  one  critical  factor  was  unanimously  not  included  in  the 
acquisition  plans.  Modification  of  COTS  items  being  unacceptable  was  not  included  in 
any  of  the  acquisition  plans  of  the  programs.  These  results  are  shown  in  Table  5. 


Not  Rated  Highly  Successful 

Highly  Successful 

Question 

A 

B 

c 

D 

E 

F 

G 

H 

I 

J 

Is  a  plan  for  upgrades/obsolescence  included? 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Is  modification  of  COTS  item  unacceptable? 

X 

* 

Is  a  warranty  from  the  prime  contractor  included? 

X 

X 

X 

X 

X 

X 

X 

X 

Is  the  contractor  included  in  government  IPTs? 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Is  CAW /tradeoff  analysis  addressed? 

X 

X 

X 

X 

X 

X 

X 

Is  TOC  used  in  tracking  costs? 

X 

X 

X 

X 

X 

X 

X 

Table  5.  Unanimous  Results 


Qualitative  Follow  Up 

This  section  is  concerned  with  the  explanation  of  the  key  success  factors. 
Interviews  were  accomplished  at  each  program  to  determine  how  these  key  success 
factors  positively  impacted  the  program.  Key  factor  analysis  identified  two  areas  that 
could  have  an  impact  on  the  success  of  COTS-based  systems.  In  this  phase  of  the 
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research,  interviews  were  accomplished  to  further  explore  the  relationships  these  areas 
had  with  program  success.  Key  personnel  from  the  rated  highly  successful  programs 
were  asked  to  identify  reasons  the  key  success  factor  led  to  program  success.  Key 
personnel  from  the  not  rated  highly  successful  programs  were  also  asked  questions  about 
the  key  success  factor.  Personnel  from  cases  A  and  C  were  unable  to  provide  interview 
responses. 

Cost  as  an  Independent  Variable.  In  the  response  to  the  investigative  question 
‘How  did  use  of  CAIV  affect  system  success?’  all  five  of  the  rated  highly  successful 
programs  identified  CAIV  analysis  in  their  acquisition  plans. 

In  case  F,  the  chief  of  the  program  management  and  operations  division  was 
contacted.  The  division  chief  related  that  the  program  was  initiated  well  before  CAIV 
became  an  Air  Force  policy.  However,  the  program  did  use  various  forms  of  tradeoff 
analysis  to  fit  the  program  within  budget.  Without  these  trades,  budgets  would  never 
have  been  approved.  The  tradeoffs  were  not  truly  CAIV,  but  were  similar  to  the  CAIV 
analysis  that  is  done  today.  In  this  program,  performance  tradeoffs  were  used  to  obtain 
an  operationally  capable  system  while  meeting  cost  objectives. 

In  case  G,  the  contracting  officer  and  business  manager  were  interviewed.  The 
technical  requirements  were  ‘soft’  or  flexible  requirements.  Flexibility  of  requirements 
enabled  the  more  affordable  COTS-items  to  be  used.  This  flexibility  provided  the  prime 
contractor  with  the  means  to  control  costs. 

In  case  H,  the  program  manager  was  interviewed.  This  system  was  not  made  up 
of  some  items  that  were  COTS,  but  the  entire  system  was  a  COTS  item.  In  acquiring  a 
full  COTS  solution,  it  was  unlikely  that  any  one  COTS  application  would  satisfy  all  of 
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the  user  requirements.  Performing  a  CAIV  analysis  for  a  full  COTS  solution  meant  that 
some  system  requirements  might  not  be  met.  The  acquisition  team  selected  the  system 
that  gave  the  best  value  by  satisfying  the  most  requirements  while  at  the  same  time 
providing  required  scalability,  flexibility,  and  technical  environment  at  an  affordable  and 
reasonable  cost.  The  comparison  of  costs  resulted  in  acquisition  of  a  substantially 
cheaper  COTS  solution.  In  this  case,  CAIV  was  used  successfully  by  not  just  trading 
performance  parameters,  but  entire  performance  requirements. 

In  case  I,  the  three  different  procurement  contracting  officers  were  contacted.  The 
consensus  of  the  group  was  that  the  CAIV  analysis  identified  threshold  and  objective 
platforms  with  key  performance  parameters  that  could  not  be  traded.  The  threshold  and 
objectives  were  the  minimum  and  maximum  of  affordability  of  that  parameter.  In  this 
case,  CAIV  analysis  was  performed  as  outlined  in  the  CAIV/TOC  Guidebook. 

In  case  J,  an  operations  research  systems  analyst  allowed  an  interview.  Although 
not  a  contracting  officer  or  program  manager,  this  person  was  in  charge  of  the  acquisition 
plan  for  the  program.  The  analyst  stated  that  tradeoff  analysis  was  used  within  the 
principles  of  CAIV  analysis.  This  tradeoff  analysis  led  to  changes  in  the  design  of  the 
system  that  allowed  different  components  to  be  used.  These  components  either  cost  less 
to  use  in  the  production  of  the  system  or  reduced  operating  and  support  costs.  In  either 
case,  the  components  selected  for  use  were  more  reliable  than  the  ones  they  replaced.  By 
selecting  the  best  COTS-item  for  inclusion  in  the  system,  life  cycle  costs  were  reduced. 
Again,  CAIV  analysis  was  performed  as  outlined  in  the  CAIV/TOC  Guidebook. 

Among  the  not  rated  highly  successful  programs,  three  of  the  programs  were 
coded  as  not  using  CAIV  analysis.  In  case  B,  both  the  program  manger  and  the 
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contracting  officer  on  the  program  were  contacted.  Both  responded  that  CAIV  was  in  its 
infancy  and  not  required  at  the  time  the  request  for  proposals  was  released.  The 
procurement-contracting  officer  in  case  D  also  stated  that  CAIV  analysis  was  not  used  at 
the  time. 

Case  E  did  require  use  of  CAIV.  CAIV  was  used  in  respect  to  design  to  cost 
(DTC)  and  life  cycle  cost  (LCC)  efforts.  A  target  price  was  also  set  for  the  program. 
Performance  goals  were  defined  that  were  influenced  by  LCC.  LCC  were  reviewed  in 
design  of  systems,  subsystems,  support  systems,  and  training  systems.  Additionally, 
significant  efforts  were  made  in  the  EMD  phase  of  the  acquisition  to  get  a  reduction  in 
operating  costs.  However,  data  is  not  in  yet  on  the  results  of  those  efforts.  In  this  case, 
CAIV  was  used  as  outlined  in  the  CAIV/TOC  Guidebook. 

In  using  CAIV  analysis,  all  of  the  rated  highly  successful  programs  traded  off 
performance  parameters  for  cost  objectives.  One  of  the  programs  even  traded 
performance  requirements  for  cost.  These  tradeoffs  led  to  a  reduction  in  cost  while 
maintaining  operational  capability.  Identification  of  requirements  and  parameters  that  are 
flexible  was  the  key  to  CAIV  success.  For  CAIV  to  work  effectively  in  a  COTS-based 
system,  requirements  must  be  flexible  enough  to  identify  tradeoffs  in  performance 
parameters. 

Total  Ownership  Cost.  In  response  to  the  investigative  question  ‘Is  TOC  used  in 
tracking  costs?’  all  five  of  the  rated  highly  successful  programs  were  coded  yes. 

In  case  F,  the  chief  of  the  program  management  and  operations  division  stated 
that  reduction  in  life  cycle  (total  ownership)  costs  was  a  goal  of  the  program.  CAIV  and 
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TOC  were  used  together  to  meet  those  cost  goals.  This  reduction  in  costs  was  attained 
through  improved  reliability  and  maintenance. 

In  case  G,  TOC  was  used  extensively  in  reducing  logistics  costs.  TOC  and  CAIV 
are  used  together  to  reduce  life  cycle  costs.  Additionally,  a  just  in  time  logistics  system  is 
being  used  effectively  eliminating  spare  parts.  When  upgrades  are  proposed  for  inclusion 
in  the  system,  obsolete  spare  parts  are  not  part  of  the  cost  analysis.  Newer  technology 
has  allowed  for  upgrades  to  be  included  that  are  generally  cheaper  than  older  systems. 
This  in  turn  reduces  total  ownership  costs. 

In  case  H,  the  program  manager  stated  that  the  Total  Ownership  Cost  (TOC)  was 
substantially  reduced  in  the  program.  He  believed  this  was  due  to  the  acquisition 
strategy.  The  acquisition  was  approached  with  the  attitude  that  lower  TOC  was  a  result 
of  smart  acquisition  planning  and  execution.  TOC  was  a  result,  not  a  goal  in  and  of  itself. 
The  advantage  of  using  a  true  COTS  system  was  that  license  fees,  annual  maintenance, 
and  labor  rates  were  all  awarded  using  a  firm  fixed  price  contract.  This  produced  a  true 
CAIV  analysis  that  allowed  for  reduced  TOC.  Again,  TOC  and  CAIV  were  used 
together  to  attain  reduced  life  cycle  costs. 

In  case  I,  TOC  was  used  in  viewing  the  operational  and  sustainment  costs  for  the 
life  of  the  system.  As  part  of  the  affordability  requirement,  TOC  was  one  of  the  criteria 
used  in  the  selection  of  a  contractor.  TOC  and  CAIV  were  used  together  to  reduce  life 
cycle  costs. 

In  case  J,  TOC  was  used  as  part  of  the  CAIV  analysis.  The  program  office  could 
not  have  used  CAIV  without  using  TOC  or  TOC  without  CAIV.  Therefore,  TOC  was 
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used  as  a  part  of  the  component  selection  process  to  reduce  production  and  operating  and 
support  costs. 

Of  the  not  rated  highly  successful  programs,  three  did  not  identify  use  of  TOC  in 
their  acquisition  plan.  In  cases  B  and  D,  TOC  was  not  used  because  it  was  not  required  at 
the  time.  In  case  E  use  of  TOC  was  not  readily  apparent,  but  it  was  used  in  the  program. 

Case  E,  coded  no,  did  actually  use  TOC.  An  interview  was  done  with  an 
acquisition  consultant  to  the  program  who  is  in  charge  of  the  acquisition  plan  to 
determine  why  TOC  was  not  identified  well  in  the  acquisition  plan.  The  acquisition  plan 
for  the  program  identified  use  of  TOC,  but  did  not  use  operations  and  maintenance  costs 
in  the  analysis  or  tracking.  The  consultant  stated  that  direction  to  the  contractor  about  life 
cycle  costs  and  TOC  was  provided  in  the  contract.  The  contract  provided  goals  for  life 
cycle  cost  and  directed  the  contractor  to  perform  life  cycle  costs  studies  throughout 
product  development.  Life  cycle  cost  analysis  was  also  provided  in  the  operational 
requirements  document  (ORD)  to  keep  operations  and  support  costs  low.  Additionally, 
the  tenets  of  TOC  and  CAIV  are  part  of  the  program.  However,  the  consultant  noted  that 
TOC  and  LCC  are  not  obvious  in  the  acquisition  plan.  Even  though  this  program  was  not 
rated  highly  successful,  CAIV  and  TOC  seemed  to  be  applied  correctly. 

All  of  the  highly  successful  programs  identified  use  of  TOC  in  tracking  costs. 
They  also  identified  that  CAIV  and  TOC  were  used  together  and  were  not  easily 
separated.  All  of  the  benefits  received  from  using  CAIV  apply  to  TOC  as  well.  TOC 
and  CAIV  together  allowed  programs  to  lower  operations  and  support  costs  and 
meet  cost  goals. 


62 


Summary 


This  chapter  provided  the  results  and  analysis  of  the  research.  The  key  factors 
were  analyzed  to  determine  if  they  could  be  considered  a  key  success  factor.  Both  CAIV 
and  TOC  were  determined  to  be  key  success  factors.  Critical  factors  were  analyzed  next. 
The  difference  in  the  amount  of  critical  factors  in  rated  highly  successful  programs  versus 
other  programs  was  2.4.  Five  of  the  critical  factors  were  included  in  all  of  the  rated 
highly  successful  programs.  One  of  the  critical  factors  was  not  included  in  all  of  the  not 
rated  highly  successful  programs.  TOC  and  CAIV  were  also  found  to  be  critical  factors 
that  were  present  in  all  of  the  rated  highly  successful  systems.  In  the  qualitative  analysis, 
interviews  were  accomplished  to  determine  how  these  factors  affected  program  success. 
Flexibility  of  requirements  allowed  for  the  tradeoffs  to  be  made  in  a  CAIV  analysis. 

TOC  and  CAIV  were  used  together  to  reduce  operating  and  support  costs.  TOC  and 
CAIV  also  allowed  programs  to  meet  life  cycle  cost  goals.  Chapter  five  will  develop  a 
theory  that  can  be  used  to  explain  the  relationship  of  TOC  and  CAIV  to  the  success  of 
COTS-based  systems. 
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V.  Findings  and  Conclusions 


Overview 

COTS-based  systems  have  been  offered  as  a  way  for  the  DoD  to  reduce  costs 
while  keeping  current  technology  in  the  hands  of  the  warfighter.  COTS-based  systems 
have  different  risks  and  problems  associated  with  them  than  traditional  military  systems. 
Currently,  Air  Force  policy  on  acquisition  strategy  does  not  specifically  address  issues 
with  COTS-based  systems.  The  acquisition  strategy  of  a  program  is  documented  in  the 
program’s  acquisition  plan.  Currently,  there  is  no  standard  guidance  for  the  development 
of  an  acquisition  plan  for  COTS-based  systems.  This  chapter  provides  a  theory  based  on 
the  preceding  research  of  how  total  ownership  cost  (TOC)  and  cost  as  an  independent 
variable  (CAIV)  can  work  together  to  affect  the  success  of  a  COTS-based  system. 
Initially,  COTS  problems  are  reviewed.  Then  the  relationships  between  TOC  and  CAIV 
are  presented.  Subsequently,  two  theories  are  presented  on  how  TOC  and  CAIV  can  lead 
to  the  success  of  a  COTS-based  system.  First,  the  use  of  CAIV  and  TOC  will  lead  to  the 
success  of  a  COTS-based  system  through  mandating  flexible  requirements.  Second,  the 
use  of  TOC  and  CAIV  can  reduce  problems  associated  with  system  upgrades.  Next,  the 
limitations  of  the  research  are  explored.  Finally,  recommendations  for  future  research  are 
offered. 
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COTS 


As  shown  in  chapter  2,  COTS-based  systems  have  certain  problems  and  risks 
associated  with  them.  Problems  in  COTS-based  systems  are  due  to  inflexible 
requirements,  technology  cycle  time,  upgrades,  and  budget.  Inflexible  requirements 
restrict  the  number  of  COTS-items  that  can  be  proposed  for  use  in  a  system.  Technology 
cycle  time  can  lead  to  problems  in  COTS-based  systems  because  the  life  of  a  typical 
military  system  usually  exceeds  20  years.  Upgrades  in  technology  can  quickly  make  a 
military  system  obsolete.  Another  problem  is  COTS-based  systems  is  upgrades  of 
COTS-items.  Upgrades  to  one  COTS-item  may  cause  problems  with  another  COTS-item 
in  the  same  system.  Budgetary  problems  in  a  COTS-based  system  come  from  the 
incorrect  application  of  life  cycle  costs.  Operations  and  support  costs  of  COTS-based 
systems  can  be  high  due  to  changing  logistics  needs  of  upgraded  items.  Risks  in  COTS- 
based  systems  are  associated  with  upgrades,  quality,  security,  and  funding.  Failure  to 
upgrade  to  the  newest  version  of  a  COTS-item  can  result  in  loss  of  vendor  support  for  the 
item.  Quality  in  a  COTS-based  system  is  risked  because  quality  is  a  subjective  measure 
based  on  the  supplier’s  point  of  view.  Security  is  a  risk  because  COTS-based  systems  are 
particularly  susceptible  to  a  trap  door  or  a  Trojan  Horse. 


TOC/CAIV 

There  seems  to  be  a  relationship  between  TOC  and  CAIV  with  respect  to 
reducing  defense  systems  life  cycle  costs.  Defense  systems  TOC  is  defined  as  life  cycle 
costs.  Life  cycle  costs  are  driven  by  requirements  ‘pull’  and  technology  ‘push’. 
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Requirements  pull  starts  when  a  system  operator  encounters  a  new  threat.  A  new  threat 
leads  to  weapons  systems  requirements  change.  Systems  are  then  changed  to  meet  this 
new  threat.  Technology  push  starts  when  new  capabilities  are  developed.  Systems  are 
then  upgraded  to  include  the  newest  capabilities.  Both  requirements  ‘pull’  and 
technology  ‘push’  increase  operational  capabilities  and  change  system  costs.  A  balance 
is  needed  between  operational  capabilities  and  system  costs.  Cost  as  an  Independent 
Variable  (CAIV)  is  the  tool  used  to  achieve  this  balance.  CAIV  is  used  to  obtain  the  best 
system  with  the  lowest  total  ownership  cost.  Achieving  the  lowest  TOC  is  made  possible 
by  ensuring  the  review  of  not  only  short  term  costs,  but  also  costs  including  research  and 
development,  investment,  operations  and  support,  and  disposal  of  a  system.  Two 
principles  are  used  within  CAIV  to  achieve  the  balance  between  operational  capabilities 
and  system  costs.  First,  a  cap  is  placed  on  system  costs.  Second,  trade  space  is  used  to 
identify  a  range  of  alternatives  in  system  requirements.  Having  this  range  in  each 
requirement  allows  decision  makers  to  identify  options  that  balance  operational 
requirements  with  system  costs. 

Analysis 

Flexible  Requirements.  The  use  of  CAIV  mandates  the  use  of  flexible 
requirements.  In  performing  a  CAIV  analysis,  trade  space  needs  to  be  defined.  This 
trade  space  is  made  available  by  identifying  key  performance  parameters.  Key 
performance  parameters  are  identified  by  setting  goals  and  thresholds  for  certain 
requirements.  By  setting  goals  and  thresholds,  the  system  requirements  are  made 
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flexible.  This  is  shown  in  the  analysis  of  the  key  factors  at  Table  3.  Each  of  the 
programs  rated  highly  successful  by  the  Scientific  Advisory  Board  used  CAIV.  All  of 
these  systems  also  have  flexible  requirements.  Although  case  G  was  coded  no  regarding 
flexible  requirements,  the  requirements  were  flexible  as  stated  by  the  contracting  officer. 
Flexibility  of  requirements  in  this  case  was  not  specifically  addressed  in  the  acquisition 
plan  because  flexibility  was  understood  to  be  included  when  using  CAIV. 

In  the  programs  not  rated  highly  successful  by  the  Scientific  Advisory  Board,  case 
E  was  the  only  one  of  the  acquisition  plans  that  identified  both  CAIV  and  flexible 
requirements.  While  this  case  was  not  rated  highly  successful,  it  may  have  still  been  a 
successful  system.  The  SAB  did  not  delineate  between  levels  of  success.  Therefore,  any 
of  the  cases  that  were  not  rated  highly  successful  could  have  been  good  programs  but  did 
not  earn  the  rating  of  highly  successful.  Case  E  did  not  include  operations  and  support 
costs  in  their  analysis  of  total  costs.  This  could  have  led  to  the  program  not  being  rated 
highly  successful. 

Another  program  not  rated  highly  successful,  case  C,  had  CAIV  and  TOC 
identified  in  the  acquisition  plan  but  did  not  have  flexible  requirements  identified  in  the 
acquisition  plan.  This  may  have  been  the  reason  the  program  was  not  rated  highly 
successful.  Without  flexible  requirements,  CAIV  will  not  work.  CAIV  requires  trade 
space  that  is  not  available  without  flexible  requirements.  Flexible  requirements  may  have 
been  assumed  to  be  present  with  CAIV  use,  but  the  acquisition  plan  did  not  address  the 
issue.  Interviews  may  have  helped  determine  the  cause  of  the  disparity.  However, 
personnel  from  this  case  were  unable  to  provide  interview  responses.  Without  interview 
responses,  case  C  presents  a  problem  with  the  results  of  the  study.  The  acquisition  plan 
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for  this  case  included  TOC,  CAIV,  and  7  critical  factors.  The  only  real  difference  in  this 
case,  from  the  highly  successful  programs  is  not  including  flexible  requirements.  Further 
research,  beyond  the  scope  of  this  effort,  is  needed  to  determine  why  case  C  was  not  rated 
highly  successful. 

Since  all  of  the  highly  successful  systems  used  CAIV,  this  ensured  that 
requirements  of  the  systems  were  flexible.  Having  flexible  requirements  allows  decision 
makers  to  obtain  the  COTS-items  that  will  maintain  a  balance  between  operational 
capabilities  and  system  costs.  In  CAIV,  system  costs  are  capped.  When  using  TOC,  total 
life  cycle  costs  are  used  in  developing  cost  estimates.  Each  requirement  will  then  be 
balanced  in  terms  of  increased  operational  capability  and  total  system  life  cycle  costs. 
This  balance  is  accomplished  while  maintaining  system  costs  that  are  capped. 

Therefore,  the  use  of  CAIV  and  TOC  in  a  COTS-based  system  leads  to  maintaining 
or  lowering  costs  while  increasing  operational  capability. 

While  the  relationship  may  not  be  causal,  there  seems  to  be  a  strong  correlation 
between  the  use  CAIV,  TOC,  and  flexible  requirements  with  the  success  of  a  COTS- 
based  system. 

Upgrades.  Technology  cycle  time  leads  to  the  need  for  upgrades  in  COTS-based 
systems.  If  upgrades  are  not  performed,  programs  run  the  risk  of  losing  vendor  support. 
Upgrades  are  also  needed  as  the  result  of  requirements  pull  from  system  operators.  Life 
cycle  costs  escalate  due  to  system  upgrades.  However,  when  using  CAIV,  system  costs 
are  capped.  Through  interviews  with  key  personnel,  TOC  and  CAIV  used  together  were 
determined  to  lower  life  cycle  costs.  By  placing  a  cap  on  systems  costs  and  ensuring 
review  of  the  life  cycle  costs,  TOC  and  CAIV  work  together  to  reduce  the  costs  of 
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systems  upgrades.  Upgraded  items  will  only  be  included  in  the  system  if  life  cycle  costs 
will  not  increase.  All  of  the  highly  successful  programs  identified  use  of  TOC  and  CAIV 
together.  Additionally,  all  of  those  cases  included  a  plan  for  upgrades.  One  of  the 
programs  not  rated  highly  successful,  case  C,  also  included  CAIV,  TOC,  and  a  plan  for 
upgrades.  The  exception  of  this  case  was  discussed  above.  A  plan  for  upgrades  using 
TOC  and  CAIV  provides  for  reduced  life  cycle  costs  in  COTS-based  systems  while 
allowing  for  system  upgrades. 

Again  the  relationship  may  not  be  causal;  however,  there  seems  to  be  a  strong 
correlation  between  the  use  CAIV,  TOC,  and  a  plan  for  upgrades  with  the  success  of  a 
COTS-based  system. 

Limitations 

Several  limitations  concerned  me  throughout  this  project.  The  acquisition  plans 
studied  were  not  all  from  the  same  phase  of  the  life  cycle  of  the  system.  While  some 
systems  were  in  the  development  phase,  others  were  in  production,  and  one  system  was 
terminated.  Different  areas  of  the  acquisition  pan  are  emphasized  during  different  phases 
of  the  system  life  cycle.  This  may  be  why  some  acquisitions  identified  a  key  factor  and 
others  did  not.  Also,  the  acquisition  plans  came  from  different  years.  While  most  of  the 
plans  were  from  1998  through  2000,  the  acquisition  plan  from  one  program  was  written 
in  1994.  If  certain  key  factors  were  not  developed  at  the  time,  the  acquisition  plans 
would  not  contain  them.  Also,  not  all  programs  were  able  to  release  their  acquisition 
plan  to  me.  The  contracting  officer  from  case  I  would  not  release  the  acquisition  plan  to 
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me,  but  did  review  the  acquisition  plan  and  provided  the  answers  to  key  factor  analysis 
for  that  program.  This  may  have  resulted  in  answers  to  questions  being  biased  on  the  part 
of  the  contracting  officer. 

The  limitation  that  caused  the  greatest  concern  was  the  definition  of  a  ‘highly 
successful’  program.  This  study  relied  on  the  Scientific  Advisory  Boards  (SAB)  decision 
that  the  programs  were  highly  successful.  However,  the  SAB  did  not  delineate  between 
the  degrees  of  success  of  the  remaining  systems.  This  is  especially  problematic  with  the 
analysis  of  case  C.  The  acquisition  plan  for  case  C  included  the  key  success  factors,  7  of 
the  critical  items,  and  all  of  the  critical  items  unanimous  to  the  highly  successful 
programs.  If  this  case  was  considered  to  be  near  the  highly  successful  range,  validity 
would  be  added  to  this  research.  However,  if  this  case  was  not  near  the  highly  successful 
range,  the  validity  of  this  research  would  be  decreased.  Additionally,  the  requirements 
for  a  system  to  be  rated  highly  successful  were  subjective.  The  SAB  report  did  say  that 
all  highly  successful  programs  “were  selected  to  represent  the  best  program  attributes  by 
both  government  and  industry  officials”  (Grant,  2000:18).  This  subjectivity  may  have  led 
to  systems  being  improperly  included  in  or  excluded  from  the  highly  successful  range. 
This  would  also  lead  to  decreased  validity  of  the  research. 

Recommendations  For  Future  Research 

As  a  result  of  my  experiences  throughout  this  endeavor,  I  have  identified  some 
opportunities  for  future  research.  With  respect  to  COTS-based  systems,  an  accurate  cost 
analysis  tool  needs  to  be  developed  for  setting  baselines  and  tracking  costs.  Engineering 
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and  support  costs  are  different  with  COTS-based  systems  due  to  the  cycle  of  continuous 
upgrades.  The  importance  of  TOC  and  CAIV  point  to  the  importance  of  controlling  costs 
in  a  COTS-based  system.  An  adequate  tool  needs  to  be  devised  that  will  help  set  a 
baseline  for  a  system  and  track  costs  accurately. 

Another  area  of  research  that  needs  to  be  developed  is  with  respect  to  the  role  of 
the  systems  engineer.  A  systems  engineer  typically  is  responsible  for  ensuring  all  of  the 
parts  of  a  system  work  together.  This  can  be  difficult  with  a  COTS  based  system  due  to 
continual  updates  of  COTS  products.  Systems  engineers  need  to  keep  abreast  of  the 
market  conditions  and  trends  that  lead  to  upgrades  of  their  systems.  They  also  need  to 
budget  for  the  upgrades  in  order  to  keep  the  system  current.  Additionally,  they  need  to 
ensure  these  upgrades  do  not  cause  problems  with  other  COTS-items  in  their  system. 

The  role  of  the  systems  engineer  needs  to  be  redefined  with  respect  to  COTS-based 
systems. 

Closing  Remarks 

COTS-based  systems  are  being  used  by  the  DoD  as  a  means  of  reducing  costs  and 
infusing  current  technology  into  systems.  However,  COTS-based  systems  have  certain 
requirements,  cost  structure,  and  risks  associated  with  them.  While  the  use  of  COTS- 
based  systems  seems  to  be  the  direction  the  Air  Force  is  heading,  the.training  and  tools 
available  to  the  people  acquiring  these  systems  needs  to  change.  The  use  of  TOC  and 
CAIV  can  have  a  significant  effect  on  the  success  of  a  COTS-based  system.  These  items 
need  to  be  included  in  the  acquisition  strategy  of  COTS-based  systems.  While  this 
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research  may  not  provide  a  full  answer,  it  should  provide  a  good  starting  point  for  those 
that  need  to  establish  policy  in  regards  to  acquisition  plans  of  COTS-based  systems. 
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Appendix  A:  Interview  Questions 


Questions  for  the  highly  successful  programs 

How  did  CAIV  /  tradeoff  analysis  lead  to  system  success? 

Was  TOC  part  of  the  CAIV  analysis? 

All  of  the  responses  included  TOC  as  a  part  of  CAIV.  Therefore,  further  questions  were 
not  needed  to  analyze  how  TOC  led  to  system  success. 

Questions  for  the  programs  not  rated  highly  successful  with  acquisition  plans  that  did  not 
identify  use  of  CAIV 

Was  CAIV  used? 

If  yes,  why  was  CAIV  not  included  in  the  acquisition  plan? 

If  yes,  was  TOC  used  as  a  part  of  the  CAIV  analysis? 

Questions  for  the  programs  not  rated  highly  successful  with  acquisition  plans  that  did  not 
identify  use  of  TOC 


Was  TOC  used  to  track  costs? 

If  yes,  why  was  it  not  addressed  in  the  acquisition  plan? 
If  yes,  was  CAIV  used  as  part  of  the  TOC  analysis? 
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